nikclayton

Mastodon: https://mastodon.social/@nikclayton

Edit to add this set of links to the posts in the series

That was an interesting 24 hours or so.

First of all, if you're one of many people who boosted or commented on the original post, or left feedback appreciating any of the open source work I have done, thank you.

After the project's financial admins shutdown discussion I spent some time reassessing my concerns, making a gut-check they seemed valid. It's still a risk to bring these sorts of issues to a public forum, and I'm relieved so many of the replies and commentary I've seen elsewhere agrees that yes, this is something to be concerned about.

I said I'd collate responses to issues I'd seen raised and reply in a single post, so that's this. I'm aware it's not an ideal format, but neither is fitting each reply in to multiple 500-character posts and hoping they federate properly so everyone sees the full discussion either.

If I've missed something below please feel free to send me a DM and I'll follow up in the next update (I'm pretty certain there'll be one, if only to announce a new git repository...)

Thank you for posting this. It is indeed concerning.

Whatever you do in the future, please consider not creating your own legal entity but instead joining one of the existing (well respected) fiscal host organizations like Software Freedom Conservancy, Software in the Public Interest, etc. They can handle all of the assets, legal paperwork, liability, etc. - https://mastodon.km6g.us/@kevin/110963590931224115

Yes, that's a very important point, thanks for raising it.

To organise my thoughts about this I started framing them as pitch for a new association. The early draft is pitch.md.

There's a great deal of overlap between what I describe there and the work other organisations do, and I would prefer to work with an existing organisation than contribute to further fragmentation.

I have already contacted

After some research those seemed to be the most appropriate three, but I may have missed some, so if you, or anyone else, can recommend other organisations that would be a good fit I'd be happy to hear it.

so, to summarize, you found a payment to a contractor suspicious because you couldn't find any code from the work, and then you talked to the contractor and found the code, so your suspicions were in fact unfounded.

And also you think it's a conflict of interest for the person who does the accounting for the org to also be on the payroll for he org... like an accountant - https://social.coop/@datatitian/110964853363902591

I see several other people have already replied to you correcting the inaccurate summary in your first paragraph.

Re the second, an accountant does not generally have sole approval authority over expenses. Consider this hypothetical. A financial admin drawing significant funds from the project receives a funding proposal from someone else. Approving this proposal would reduce the amount of money available for them, so they decide to reject the proposal.

There's the conflict.

I am not certain that the first two counts are suspicious. Payment was made for work on three issues. It sounds like the person was legit, they just didn't finish the work after being paid?

The second is just silly. Project Management is a job title and job description. It is explicitly what they mean by “someone unfamiliar with your project” will recognize.

Invoices don't list specifics. There should be internal docs for that. Those docs often aren't public.

If the project (hypothetically) hired me, I'd appear on the invoice as “Policy & Regulatory Analysis”. You would NOT list the different policy issues involved on the invoice. That would be documented internally for many reasons. What the accountants need to know is “what was this money used for?”

I do not work on IT regs, so I cannot speak to specifics, but I do think that it's better to ask an attorney or accountant to look first before raising suspicions. - https://med-mastodon.com/@UncivilServant/110964976957986119

It's not “didn't finish the work”, it's “didn't do the work”. The payee may have worked on the issues in private. But if they don't push the work somewhere public, create PRs from it, link to it, etc, then it is indistinguishable from not doing the work.

As I wrote in the initial post, there was no public indication any work had been done on these issues.

In your “Policy & Regulatory Analysis” example, as an open source project, I would expect to see, at minimum, a link to the work you produced as part of that analysis. That would be “sufficient detail to make it clear what the collective is paying for”, per the OpenCollective guidelines.

I would also expect any agreements the project makes with third parties to clearly set out what the expected deliverables are.

This works even for problems where the shape of the solution is not known beforehand.

For example a bad agreement would be “Solve problem X”. After a month the payee comes back and says “Sorry, I tried lots of things, I couldn't solve it, I've got nothing to show for it, here's my invoice.”

A better agreement would be “Solve problem X. If you can't solve X, deliver detailed descriptions of the approaches you took and why they don't work, so future attempts to solve this problem can build on what you learned.”

That's a lot of book keeping and managing overhead tbh. [...]

My greatest fear of launching monetary-focused foss products is this management overhead which compared to the rest of the industry just feels so unworthy. The monetary incentives just don't add up so I can definitely see how easy it'd be to get lost here. - https://fosstodon.org/@wraptile/110965923454931957

Yes, I agree. Introducing expenses, payments, grants, etc in to a project can add a lot of overhead.

Projects should not do this unless they have the necessary people, time, and expertise to do this properly.

In other words, don't adopt processes you're not staffed to meet (a variant of a rule I described in Rules for a healthy on-call rotation as “Don't set SLOs you are not staffed to meet”).

You’re expecting the financial admins to do their work for free. If anything, you’re making yourself look bad here, and not the Open Collective. - https://eupolicy.social/@whvholst/110965996322449276

I'm definitely not expecting someone to work for free. If the only way someone can contribute to an open source project is to be paid, and the project has funds to pay them that's absolutely fine.

That does not absolve them of the responsibility to do things properly, and with accountability.

Finally, this thread:

I've said this in the past, but maybe today it's more important than it's been in a very long time to tell you.

Thank you everyone who's been supporting #Tusky over the years on open collective. You've been supporting my disabled ass through so much, and I always knew I had a little bit of money at the end of the month, even when I was the most sick. ($100 when sick, $175 when able to work)

Me and Conny worked out this agreement back sometime after I started the open collective for the project (early 2019?). I've mostly been providing support, and project managment at some points. Organizing people and texts. A lot of invisible work. . I'll always be eternally grateful for everything you've done for us and me. - https://rage.love/@maloki/110969436075202543

If this shit turns out to be a techbro devaluing the invaluable (and largely taken on by women) glue work Yet Again I will yeet my chair through my office window. - https://comicscamp.club/@nebulos/110969752596667389 (direct link broken at request of the original poster)

i mean it partially is. - https://rage.love/@maloki/110969756892267642

It really partially isn't.

I was careful in my original post not to name anyone directly, I was expecting a collective response from the Tusky project. Since Maloki has come forward in public I won't anonymise comments from her in the future. I will continue to redact the names of anyone else in the project unless they also respond separately from the project.

Having dealt with that, there's quite a lot to unpack here.

First, @maloki@rage.love is the financial admin quoted in the original post, who initially insisted the discussion be shut down, and as is apparent from these posts, also the recipient of a substantial fraction of the project's funds.

That conflict of interest should be noted.

Second, an uncharitable reading of this thread — and while I continue to assume these issues were mistakes and not a deliberate attempt to hide wrongdoing, the needle on that dial is starting to wobble — could see this as an attempt to derail this conversation away from the concerns I raised, DARVOing the discussion.

Let's not do that, please.

Third, I think a reasonable reading of “$100 when sick, $175 when able to work” is the project was paying you during months when you did no work.

That seems to be supported by invoices like #30003, “Sick-pay June-December”.

If that's a mis-read of that line I'm happy to be corrected.

If that's not a mis-read then I have a host of other questions, starting with, was the project effectively paying you a salary as an employee?

I believe you're in the UK, so even if you think you were an independent contractor, IR35 probably applies (not an accountant, but I used to run a contractor business in the UK, so I have experience with this) and this probably should have been treated as employment income.

  • Was that legal?
  • Was the project paying you at least minimum wage (GBP 10.42 an hour)
  • Was the project paying appropriate taxes and employee insurance contributions?
  • Does the project have an employee-like arrangement with anyone else?
  • Why wasn't the process for using OpenCollective to pay employees (Employment & Benefits – Open Collective Foundation) followed?

Fourth, your comment “sometime after I started the open collective for the project” prompted me to go and look at those first invoices, at Tusky · Expenses – Open Collective

Of the eight invoices on that page. two are for Tusky project expenses (domain name, stickers), one is payment to a translator, for a total of USD 129.53. The rest (USD 960) are payments to you.

An uncharitable person could think the project did not actually need the funds, and that you started the project's OpenCollective to provide a front that would allow you to collect money hoping that it wouldn't be noticed.

A charitable person could assume that is not the case, but still agree that the optics of this are terrible, especially after you repeatedly tried to shut down any discussion of the project's invoices.

Fifth, “I've mostly been providing support, and project managment at some points. Organizing people and texts. A lot of invisible work”.

Undervaluing people's contributions is an ongoing issue, and a bit of a hot-button topic. If you're new to this I recommend Being Glue — No Idea Blog.

So let me make a couple of hopefully non-controversial statements:

  1. A quid pro quo for taking money from the project for “invisible” work is it's on you to make sure the work becomes visible.

  2. If you can't describe what you're doing for the project, how is the project going to replace you when you step down?

I know what that work looks like (I'm aware of the “invisible” / “looks like” irony), I've spent years in previous roles doing project management work, I've had the good fortune to be mentored by some truly excellent project managers, and I've worked to get people successfully promoted by ensuring that the impact of their glue work was recognised.

And I've had the opportunity to learn from some epic screw ups of my own.

So here's a short list of things you could have held yourself accountable for doing:

  • Curating the list of open project issues (as in #3240 for example)
    • Following up on incomplete bug reports
    • Checking if new releases solved the reporter's problem
    • Making sure that bug reports were not being ignored
    • Applying consistent labels to make them easy to manage
  • Improving the onboarding experience for new contributors
  • Planning a release calendar, or any form of release scheduling
  • Writing the release notes for each release
  • Proactively adding new contributors to the GitHub repo
  • Finalising a complete expense policy
  • Finalising a handover document to prepare for your departure (document promised since early 2023)
  • Following up on stalled PRs
    • Had the submitter run out of time?
    • Did they not understand the review feedback?
    • Were they waiting for review and the project had slipped up?
  • Creating GitHub issues to track problems reported by users to the @tusky account
  • Communicating the state of ongoing activities to the rest of the project contributors
  • Tracking requests for comments on proposal documents
  • Liaising between the Tusky project and other Fediverse projects
    • I'm aware of your overlap with the GoToSocial project. You seemed actively hostile to any communication with the folks at Mastodon GmbH.
  • Outreach to sites comparing different Mastodon clients, ensuring their info about Tusky was accurate
  • Keeping the OpenCollective page up to date with project news
    • I asked for this multiple times — it could have been as simple as copying the release notes in to an announcement after each release.
  • Regular check-ins with each contributor to the project; how are they doing, do they have the support they need, do they have any issues that need addressing, etc
  • Responding to inaccurate or mistaken reviews on Google Play

Not all of these things were being done by the project, but the ones that were were all being done by other people. Framing your contributions as non-specific “providing support, and project managment at some points. Organizing people and texts. A lot of invisible work.” is a very effective way for other people to get the impression that you must have been involved in these things, denying the credit to the people who were actually doing the work.

I'm not for the moment suggesting you have to do all of these things. But doing one or two of them effectively would have reduced the burden on other project contributors who weren't, it seems, being paid a salary.

I'll also highlight the distinction between being accountable for something and being responsible for something.

  • Accountable – I will make sure this gets done, I will find the right people to do it
  • Responsible – I will be the person actually doing the work

You may not have been the person who could be responsible for a piece of work, but you could have been the person who was accountable for a piece of work.

The concrete things I think you did in the 8 months I was involved (often after you were prompted to do them) were:

  • Creating chat rooms
  • Organising and chairing the first team meeting (after multiple people called for one to happen)
    • Subsequent team meetings were completely handled by [redacted], who scheduled them, managed the agenda, and chaired the meetings
    • That was helpful, the first meeting was run well, and [redacted] is continuing to do an excellent job
  • When @connyduck's private message stepping down from the project was leaked I suggested the project post an announcement, and posted a draft for discussion. You wrote your own announcement and posted that, also updating OpenCollective for the first time in more than four years.
  • Occasional comments on the chat groups or on discussion documents, no different from any other contributor.
  • An attempt to form a more formal support group with [redacted] that, to the best of my knowledge, didn't achieve anything

You may also have been doing some customer support from the @tusky account at some point before early 2023. It's difficult to tell because we didn't indicate who was making each post from the account. I'm confident you did very little, or none of that, from the time that I was also posting from the account, because I would have seen it.

Sixth, most of this is the project's problem. If the agreement (can we see it?) did not state what you were expected to do beyond vague “project management” then it becomes a lot more difficult for the project to say “hey, please can you do some of the things you're being paid to do”.

But recognising that issue, and working to correct it is a responsibility I would expect you to share.

And trying to shut down any discussion is still a terrible look.

Edit to add this set of links to the posts in the series

I discovered severe lapses in how the Tusky project's donations (received via OpenCollective) were being handled. When I reported those to the project's private “Tusky Contributors” Matrix channel the financial admins tone policed the feedback, refused to engage with the concerns I and others raised, and demanded the discussion be stopped.

That's an ethical boundary I'm not going to cross, and I'm not interested in collaborating with people who have crossed that boundary and are not prepared to walk back.

Therefore I am stepping back from the Tusky project with immediate effect, and I recommend anyone who is currently donating to the project via OpenCollective reads the following, and makes their own decision on whether or not to continue financially supporting the Tusky project.

I do not have a conflict of interest; I have not taken money from the Tusky project in the past, I do not have any open expenses with the project now, and none of my income is remotely related to the Tusky project or any competing projects.

I'll be reading any replies with questions, collating the questions directed at me, and then writing a single response addressing all of them (or one per day if the questions run over multiple days).

Why were you looking in to this?

@connyduck's recent departure from the Tusky project exacerbated an ongoing issue with the ownership of project assets (domain names, website hosting, Google Play publishing account, etc). They were owned by him, sometimes entangled with his personal accounts, and would need to be transferred somewhere new so the project could continue, and to allow him to completely disengage from the project.

After discussions with other contributors I started researching options for creating a separate legal entity to own these assets on behalf of the project.

The research showed that whatever specific form the entity took it may involve some people taking on legal liability for the project's finances.

To determine what the impact of that might be I started looking into the Tusky project's OpenCollective history to see what sort of things the project was paying for, how much was being spent, what oversight there was, and so on. This would let me recommend alternatives that matched how the project was currently doing things, and highlight anything we would need to change.

What were the lapses?

OpenCollective's policies on how projects should record expenses are very clear (Invoice and Reimbursement Examples – Open Source Collective, emphasis mine):

Invoices need a clear description of services rendered and provide sufficient detail to make it clear what the collective is paying for. This is because this description may be checked by our accountant for compliance. That means that someone with no knowledge of your project, and only a limited knowledge of open source, should be able to understand what was done from the description.

Many Tusky project invoices do not do this. And in some cases they show payments for work that was demonstrably not done.

The first invoice I noticed this on was Invoice #125597, which is for:

Work on 'bookmark tab' feature (#2368); Work on 'copy hashtags into reply' (#3013); Work on 'positional substitution format' (#3297)

That's from February this year, it's now late August.

There is no publicly available evidence of any work on those issues.

  • The payee's GitHub repository (GitHub – SomaRasu/Tusky) had no indications of any work done:
    • No branches for the work
    • No commits relevant to the work
    • No PRs proposed with that work (from either this repository, or any other)
  • The three mentioned issues were still open, #2368, #3013, #3297
    • There were no comments on those issues from anyone
    • There was no indication any work was underway to resolve those issues
  • The payee had not asked for help with these issues in any Tusky public or private Matrix channels I'm a member of, or public channels I searched through

Note: After initially raising this issue I then spent approximately two hours on 2023-08-23 and 2023-08-24 mentoring the payee to fix issue #2368, and submit a PR. They did, which I reviewed, approved, and merged. The other two issues remain open at the time of writing.

Important: I do not think any of this is the payee's fault, and have told them this. I believe the project did not provide them with sufficient support, and when it became clear they could not complete the work the project could have paused the engagement, or asked one of the other contributors to assist.

Further review showed other invoices that did not “provide sufficient detail to make it clear what the collective is paying for” and allow “someone with no knowledge of your project [...] to understand what was done from the description.”.

For example, Invoice #136617. You will find many more examples at Tusky · Expenses – Open Collective.

Why is this a problem?

Obviously, there is the violation of the OpenCollective policies.

For me, it is also a violation of the project's ethical duty to responsibly and transparently handle donations and payments.

Transparency in how people are selected to receive funds, and how much to pay, is especially important in an industry rife with bias, and where women are regularly underpaid for their work.

And due to the nature of the Fediverse, Tusky attracts funding from people who have been marginalised both historically and today. I think our duty of care to be responsible and ethical stewards of the funds is even greater because of this.

Finally, a large red flag I subsequently discovered is that one of the members of the financial team managing the OpenCollective donations is a regular recipient of funds from the project. Of the USD 3,908.56 paid out by the project in the last year (2022-08-25 to 2023-08-24) (Tusky · Expenses – Open Collective) they have received USD 1,830.00 (47% of the total).

I am not suggesting those invoices are illegitimate, or the work they paid for was not done.

But I do think good financial stewardship, as well as straightforward common sense, would have recognised the obvious potential for conflict of interest, and ensured that anyone who was so dependent on the project's funds would not have any say in controlling those funds.

Raising the issue

I first flagged Invoice #125597 as a concern in a message to the private and invitation only “Tusky Contributors” Matrix channel on 2023-08-21 (Monday).

The initial response from other Tusky core contributors noted they couldn't find any information about the invoices in question, and agreeing, on the face of it, the work described in these invoices was not done.

The Tusky financial admins then responded, noted that this was for work done under a separate contract, tried to derail the conversation by raising their own expenses, and attempted to blame me, claiming that since some of my first opensource contributions to the project overlapped with work that this person under contract was supposed to be doing, my contributions caused a problem with their work

As a first time contributor to the project at the time I was unaware of this, because the work they were supposed to be doing was completely undocumented and there was no public indication that were working in the same area.

This raised more questions. In particular, I could find no traceable work from the person under contract in the Tusky project; no commits, no PRs, no issues, no comments on issues, no questions asked in the public or private channels that I'm a member of, etc.

I asked what work was actually done, flagged the risk that without more context this could appear to be fraud, and was very clear I thought this was a straightforward mistake, writing (emphasis in the original):

To be super clear — I'm 100% not saying it is fraudulent. This could be as simple as “Some work was intended to be done, it wasn't, other work was done instead, and the expense description was not updated”.

But we should be able to explain what that work was, and we should have processes in place so that this does not happen again.

The admins:

  • Told me to stop asking questions
  • Said that being required to show proof of work would call their own invoices in to question
  • Insisted this was an issue that only the admins needed to know about

Other contributors continued to question this.

In response, the admins:

  • Swore, repeatedly
  • Insisted the contracts and general process would not be available to contributors

The discussion deteriorated from there, the admins refused to answer direct but reasonable questions, insisted that the conversation should stop, and described questions as “entitled”.

At this time my specific unanswered questions are:

  • Does the project have agreements in place with anyone else?
    • If so, with who, and what are the agreements?
  • Does the project currently plan to have agreements in place with anyone else?
    • If so, with who, and what are the agreements?
  • If it is technically possible, will the project amend the notes on previous OpenCollective invoices so they meet the letter and spirit of the OpenCollective policy?
  • Will the admins put in place a formal expenses and grants policy, with a public review period before adoption?
  • Is the project paying contributors an equal equivalent hourly rate?
    • If yes, what is the rate?
    • If no, how is the effective hourly rate determined for a contributor?

If the admins had:

  • Acknowledged the concerns
  • Agreed to investigate what happened
  • Agreed to put controls in place to prevent it happening in the future
  • Agreed to not processing any further expense claims until those controls are in place

then I wouldn't be writing this now. They didn't, so I am.

Could this be an innocent mistake?

Yes. I think the Tusky financial admins have made a number of serious, but correctable mistakes. I do not think anyone set out to deliberately defraud the project, or its donors, and I was clear about that during the discussion (see earlier).

However, instead of correcting those mistakes when the issue was raised they doubled down on them.

Their position is secret agreements in place to pay out funds from the Tusky OpenCollective are OK, and project contributors and donors should not be able to discuss these agreements before, during, or after the period they run for.

Some admin quotes from the discussion:

Can You stop? [...] This is an admin issue. The admins had agreements in place with people involved.

[...]

these conversations aren't going to be for and open to the contributors.

[...]

it's not a question that necessarily should be posed in the contributors channel

[...]

the contracts and agreements we make with people is not for public consumption, and I don't believe they have to be, which is why I was very thorns out on this question. You don't need to know any details of the agreement whatsoever besides “this was a past agreement and it got paid for”.

They also said:

we've worked on improving the process

But, when asked, refused to explain what those improvements were.

As noted earlier, other project contributors also expressed their concerns in the discussion, with comments including:

to me it also looks very odd. [...] in this case it does really look like payment for work that wasn't done. I think big point of OC is transparency and this does not look good for us. [...] I agree that we should be transparent and should have a record of agreements

[...]

That said, if it doesn't comply with OC, it can't be allowed to continue to happen like that. Especially not if Tusky gets a legal entity. With likely a legally liable treasurer. Also, especially not if the original core decision makers are also stepping down.

Who are you? Why should I care about your opinions?

I'm Nik, I've been contributing to open source projects since the early 1990s. I've run large open source events with a large financial outlay (I was one of the co-founders of BSDCon Europe back in 2001). I've started open source projects, joined open source projects, been given stewardship of existing projects as maintainers move on, and handed stewardship of projects I've started over to other people.

I've been contributing to the Tusky project since December 2022. My first contribution improved the FAQ (#16). Since then I have contributed more than 150 PRs to the project. If you've been affected by bugs like:

  • The swipe sensitivity between tabs being too high (fixed: #3148)
  • Losing your place when you press “Load more” (fixed: #3000)
  • Missing notifications (fixed: #3700 (and many others))
  • Images getting “stuck” when trying to zoom or swipe between them (fixed: #3894)

then I'm the one who fixed them.

I've also improved the app's accessibility including #3003, #3121, #3272, and #3248.

As @connyduck started reducing his involvement with the project I took on the responsibility of producing new releases, managing the releases for Tusky versions 22.0 and 23.0. That includes running the release process, writing the release notes, managing the beta programme, responding to user bug reports, and so on.

At the time of writing the Tusky OpenCollective site describes me as a “Core contributor”. Per the Open Collective documentation “Core contributors show up in the Team section of your page and can create events, but can't change settings or approve expenses.“, so this gives me no special access to project accounts or contracts.

For the last several months all (or very nearly all) of the posts or replies from the @tusky@mastodon.social account have been from me. And a cursory review of PRs over the last few months will show the majority of them are reviewed and merged by me as well.

In short; I know open source. I get work done. I know what good project governance looks like.

Why are you not naming names?

I think this is a collective failure of the project's financial admins, not any single individual, so the names of the individuals involved is not directly relevant.

I do not want to encourage a typical Internet pile-on of them, and any questions you have for them should be directed to the Tusky project's account, @tusky@mastodon.social (I am no longer monitoring or posting from that account).

By necessity a handful of the links above clearly show e.g., who submitted an expense, or was responsible for an approval. Again, please do not pile-on to those individuals, but send your questions about the project to @tusky@mastodon.social.

Do you have any more proof?

Yes. I have the chat logs of the “Tusky Contributors” Matrix channel that I am a member of, and where this issue was raised and dismissed.

I also have copies of the meeting minutes from the handful of Tusky contributor online meetings that have happened this year.

I will release the unredacted logs and/or meeting minutes if the the project's financial team agree, or those people make public statements that contradict what they said in private.

What's next?

I doubt the Tusky project financial admins will change direction. If they publicly commit and follow through on a transparent financial policy that meets the OpenCollective guidelines then I am prepared to rejoin the project and continue contributing. But if I expected that to happen I would not have written this.

If you are a financial contributor to the project I hope this has given you enough information to make an informed decision about where you send your money in the future.

Some Mastodon server owners are forming member-led cooperatives to host their online communities. I have been following CoSocial Community Cooperative as one example Social.Coop is another.

I think there may be space for a Mastodon app run along the same seven cooperative principles.

  1. Voluntary and open membership
  2. Democratic member control
  3. Member economic participation
  4. Autonomy and independence
  5. Education, Training, and Information
  6. Cooperation among Cooperatives
  7. Concern for Community

This incident has prompted me to investigate starting a project to do just that. More information soon.

Some people will cheerfully tell you that Mastodon is a different kind of social network because it doesn't have an algorithm.

Other people will angrily rail against this, insisting that “sorted in chronological order” is an algorithm, it's the only algorithm, it can't be changed, therefore Mastodon is bad because it doesn't provide any user controls.

Every now and then I get sufficiently annoyed by dumb takes about this to do a write-up. This is one of those times.

xkcd: Someone is wrong on the Internet

xkcd: Duty Calls

Go on then, tell me how people are wrong on the Internet

Both the positions above are wrong because they're lumping two completely different algorithms under the word “algorithm”.

Those algorithms are:

  1. How does Mastodon select the posts that will appear in your home timeline?
  2. How are the selected posts displayed?

The “Mastodon has no algorithm” crowd are actually saying “Mastodon has an algorithm, but it's fully controllable by you, nothing will show a post in your timeline that you did not explicitly ask for”.

As best as I can tell the “Mastodon absolutely has an algorithm” crowd don't care about algorithm #1 at all, and are focused on algorithm #2, insisting that “chronological ordering” is an algorithm, so that they have something to be angry online about.

There's a third algorithm too, How is an individual post on the home timeline displayed?, but people hardly ever talk about that one.

Selecting posts for your home timeline

This is the algorithm that actually matters.

You don't need to read the Mastodon code to understand how this works, you can deduce it from observation.

First:

  • Include all the top-level posts from people you follow
  • Include all the replies from people you follow
  • Include all the boosts from people you follow
  • Include all the posts containing hashtags you follow

Then:

  • Remove replies if the user has requested that
  • Remove boosts if the user has requested that
  • Remove matching boosts if the user has enabled the “Group boosts in timelines” settings
  • Remove posts from users who are actively muted
  • Remove posts from users who are blocked
  • Remove posts that match the user's filters to hide

That's at least two different ways the user can control what is initially chosen for their home timeline (users and hashtags) and at least six different ways the user can then control whether those posts appear on their timeline.

Displaying posts for your home timeline

The sort algorithm barely matters. It's “sort chronologically, newest first”. This is not modifiable by the user in Mastodon.

Mastodon does provide additional controls for the algorithm that determines how each post within the timeline can be displayed.

You can choose:

  • Whether to show media in general
  • Whether to show media that the poster marked as “sensitive”
  • Whether to automatically expand posts that have a content warning

And for posts that have been filtered at the “warn” level the user can, on a post by post basis, choose to view that post.

So, again, there's an algorithm for how each post is displayed, and the user can make choices that affect how that algorithm behaves.

Caveats

A few caveats about the above because this is the Internet and people will nit-pick to death.

First, it's describing the current observable behaviour of the Mastodon server and default web client.

Second, other clients (both web, and apps) for Mastodon servers exist, and some of those provide further controls on the presentation of your timeline.

For example, the Phanpy web client can

  • Display boosts separately to the timeline, in a carousel
  • Shows conversations in a threaded view grouped together in one block in the timeline

See Phanpy subtle UI implementations for more on that, including screenshots.

Third, Mastodon-like servers exist; federating with Mastodon servers, supporting the same basic API, but providing their own additional controls over what posts appear in your home timeline.

For example, the glitch-soc server implemented fine-grained post filters before Mastodon did. So for a while, users on a glitch-soc server had more control over their timeline than users on a Mastodon server.

In summary

  • Mastodon, and Mastodon-like servers, absolutely have an algorithm they use for selecting the items that make up your home timeline
  • The specific client you're using to access your Mastodon server might also provide its own algorithms for displaying the timeline, which you can adjust
  • Unlike other social media services (Twitter, Bluesky, Threads, etc), a Mastodon server is extremely transparent about the algorithm selecting the posts for your home timeline, and gives you full control of the parameters used in those decisions.
  • Ignore anyone arguing “Mastodon does not have an algorithm”, they lack critical thinking skills
  • Ignore anyone arguing “chronological order is an algorithm”. When it comes to social media “How are the posts you see chosen?” is a much more important question than “How are those posts ordered?”. If that's what they're focusing on then they're probably just looking for a reason to be angry about something online.
  • “I would like Mastodon to provide a new option X, which would affect how posts are chosen for my home timeline” is an interesting discussion to have. But that would require careful thought, and most people arguing online don't have time for that.

Prompted by the questions in https://hhmx.de/@nick/16376, the answer's too long to fit in a post.

Doing some monitoring how a original #Mastodon server responds, i never (!) saw the described link header! The response may include no link header or a header like this: Link: ; rel=“next”, ; rel=“prev”

You should get a Link header from any API response that returns a page of data, like https://docs.joinmastodon.org/methods/notifications/#get.

[Almost; in testing this I just discovered an edge case, which I've reported as a bug at https://github.com/mastodon/mastodon/issues/25495]

An API response for a single item, like https://docs.joinmastodon.org/methods/notifications/#get-one won't include the Link header.

I just tested this with https://restfox.dev against my mastodon.social account (https://github.com/tuskyapp/Tusky/pull/3610/files?short_path=00c65c2#diff-00c65c2349f395f9f9b78b0fe84d3638b78f3ef4abf98294364b5c4d0ef32473 has details on how to do this).

On my research i saw something in the #Tusky source, that looks like Tusky as clients seems to generate a link header? If so, why?

https://github.com/tuskyapp/Tusky/blob/develop/app/src/main/java/com/keylesspalace/tusky/components/notifications/NotificationsPagingSource.kt#L107-L178

That function needs to return a page of results as an HTTP Response. The code that calls that function unpacks the Response and parses the contents of the Link header, so the header must be present.

To figure out what page of results to return one of the parameters to getInitialPage is the ID of the item (in this case, a notification) that should be at the start of the page. This is the params.key value.

The Mastodon API lets you make requests like “Get me the page immediately after ID X” or “Get me the page immediately before ID X”.

But it does not let you make a request “Get me the page that includes key X”.

So the code has to do that itself, starting around line 134. It does this by making two API calls. One to retrieve the notification with ID X, and one to retrieve the notifications immediately after that.

If those calls succeed the code has two values to operate on; the notification, and the page of notifications immediately after it.

Remember that this function needs to return a single page of notifications. So it creates a fake page which contains both values. This fake page needs a Link header (because the code that calls this code expects a Link header).

So that's why it constructs one in this case.

This code is only called to get the first (or initial) page to show the user. After that the pages above and below are loaded with single calls to the Mastodon API, and return the Link header without any alterations.

Besides of details, if the header should be named “Link:” or “link:“,

HTTP headers are case-insensitive (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers) so you can use either.

Tusky 22 now sometimes reports an array index out of range (-1) or similar (taking this from my memory, for exact message, i have to check it again).

That would be a bug, a detailed report with reproduction steps would be appreciated.

During development i also saw Tusky sending requests for notifications in an endless loop... wondered why.

I suspect the min_id / max_id handling in your server is not correct.

And... could someone clarify the difference between sinceid and minid? If Mastodon will handle this differently?

sinceid = String. Return results newer than this ID minid = String. Return results immediately newer than this ID

What should be the difference between “newer” and “immediately newer”?

Suppose the user has 10 notifications, numbered 1 through 10.

  1 2 3 4 5 6 7 8 9 10
  ^                 ^
  |                 |
oldest            newest

They make the request:

/api/v1/notifications?since_id=5&limit=2

This means they want two (limit=2) notifications that are newer than notification 5 (since_id=5).

The server returns notifications 9 and 10, as these are the two newest notifications.

Think of since_id as “Start at the newest notification, count backwards limit notifications, and return from there”.

Same setup, but the request is now:

/api/v1/notifications?min_id=5&limit=2

This means they want two (limit=2) notifications that are immediately newer than notification 5 (min_id=5).

The server returns notifications 6 and 7, as these are the two notifications immediately newer than 5.

Think of min_id as “Start at min_id, then return the next limit notifications”.

Given that min_id was added to the API after since_id was my working theory is that this was a design flaw in the API, which min_id fixed after the API was first released.

Under the hood we're updating the Tusky design to make the code easier to work with and reason about, particularly with how timelines are handled.

A timeline is any list of posts. What you see on the Home tab is a timeline, so is the Notifications tab, Local, Federated, search results, trending tags, all of it.

This is a big task, and trying to do it to all timelines at once would be very difficult. So the work is broken up to focus on one or more timelines at a time. This also means that if there's anything that turns out to be a bad idea we can fix it on the affected timeline before it affects everything.

The timeline on the Notifications tab is the first one to get these changes.

So what's changed?

Missing posts

There have been occasional reports about posts missing from timelines (https://github.com/tuskyapp/Tusky/issues/3505, https://github.com/tuskyapp/Tusky/issues/3320).

This is fixed with the re-write. As I say, initially for Notifications, but on all timelines in the next major release.

No more “Load more”

The “Load more” list entry you might see as you scroll through your notifications? That's gone. Notifications now load automatically as you scroll through the list.

You can jump to the top of the loaded list by tapping the icon in the tab header (that's not new in this release). v22.1 will add a menu item to jump to the most recent notifications, even if they haven't been loaded yet.

Remembering the reading position

Your “reading position” is remembered. If you leave the tab and come back (even if you fully exit the app) you'll be restored back to the notification you were last reading.

If that notification no longer exists (maybe you dismissed it, or changed your filters so it's no longer visible) then Tusky tries to put you as close as possible to that position.

Better error handling

Your server might not be available. Maybe it's down for maintenance, or your phone doesn't have Internet connectivity. Tusky now handles this more gracefully, with better options to recover.

If you're scrolling through notifications and the next set fail to load the current set remain, and you'll see (a) more detail about the error, and (b) an option to retry. The error details are important, as they give you a better idea of what the problem is and how you might fix it.

If you interact with a post in the notifications tab (boost, favourite, bookmark, etc) and that fails you'll also be told why, with a “Retry” option.

Previously those failures could be invisible, and lead to user feedback like “I bookmarked this post, the icon changed, but it's not in my bookmarks”. That could happen because the “bookmark this post” request was sent to the server but the server didn't receive it. Tusky optimistically updated the UI to show the bookmark had succeeded, but didn't update the UI when it failed.

This was me, in https://github.com/tuskyapp/Tusky/pull/3159.

Tusky v22 changes how Android notifications are displayed.

To understand why the behaviour is the way it is it helps to know a little bit about how Android allows apps to send notifications. Strap in, this gets a little long.

Tusky notifications are sent to Android notification “channels”. Tusky creates one channel for each type of Mastodon notification you can get (boosts, favourites, mentions, etc).

Each channel is associated with a Tusky account, so if you're logged in with two accounts you'll have two channels for boosts (one per account), two channels for mentions, etc.

You can see all of this in “Account preferences > Notifications”.

Each channel has its own controls for how notifications appear. Do they pop up on screen? Do they make a sound? Do they ignore do-not-disturb, etc? That's so you can choose to get a sound if someone mentions you, but no notification at all if someone new follows you (for example). All of that is also controlled from “Account preferences > Notifications”.

Android apps can not see what the user has set their notification settings to.

This is so that a malicious app can't refuse to run unless you allow its notifications to make a sound — apps can't “blackmail” you if you disable their notifications.

Android imposes two other limits on how apps can send notifications.

First, an app can only have about 50 notifications active at once. It's “about 50” because different Android devices might set that limit differently, and there's no way for an app like Tusky to find out what the limit has been set to, or what will happen if it posts more notifications than that.

Second, if an app posts too many notifications too quickly Android may silently drop those notifications, and the user will never see them.

How does all that affect how you use Tusky?

Better summaries means more alerts

Previously, all your Mastodon notifications would appear as a single collection of Android notifications with a “summary” notification that collapsed them all.

You could expand the summary notification to see the individual notifications.

It didn't matter what type of notification they were (follow, boost, mention, etc) they all got summarised together.

Turns out that caused a bunch of bugs, which we've squashed. For example, having just one summary notification meant that it could be sent to a channel that you have muted. And then you'd never see them.

That's fixed — notifications always go to the correct channel.

So now you'll see multiple summary notifications. One summary for all your “mention” notifications, one for all your “followed you” notifications, one for all your “boosted your post” notifications, and so on.

This means you can choose to e.g., swipe to dismiss all your “boosted your post” notifications, while leaving the “mentioned you” notifications intact.

However, if you receive a lot of notifications you will probably find you're alerted more than you used to be. The old behaviour was a bug. Tusky was incorrectly hiding some notifications. Now it's not.

You can always control your notification channel settings in “Account preferences > Notifications”. You may want to check that out and either disable some of them, or switch some of them to “silent”.

Older notifications make way for newer notifications

Tusky now limits the maximum number of notifications it will display to stay under that 50 limit I mentioned. If you have more than 50 active notifications Tusky will drop them, oldest first. They still appear on the Notifications tab, they just won't be created as Android notifications.

Suppose you have 30 active Tuksy Android notifications, and receive another 30 new Mastodon notifications. Tusky will remove the oldest ~ 10 from the existing Android notifications and then show the 30 new notifications and the remaining 20 of the previous ones.

Notifications are created more slowly

Tusky will create those 30 new Android notifications more slowly. Previously it would create them as quickly as possible, which ran the risk of them being silently dropped by the device. Now it creates them roughly once per second.

If you get a lot of notifications all at once you'll see this when the notification icons flash briefly on the Android status bar at the top of the screen.

Synchronised with other apps

Mastodon apps like Tusky — or your server's web interface — can also sychronise the ID of the “most recently fetched” notification with your Mastodon server.

So if you have your Mastodon home page and Tusky open at the same time (or Tusky running on two different devices) a given notification will only appear in one place, so you don't have to dismiss the same information in two different apps or devices.

I did the work here.

Along the way we discovered lots of ways that not-Mastodon-but-claim-to-follow-the-Mastodon-API servers are not quite compatible with documented behaviour.

And some problems with Mastodon itself, like https://github.com/mastodon/mastodon/issues/25245.

We also found and fixed some nasty bugs if you were logged in with multiple accounts; Tusky could write the data from one account to the database for another account. If you were on the beta track and suffered through the period where you would get dumped at the very beginning of your notification history then thanks for bearing with us while we found and fixed the problem.

Tobi (https://goblin.technology/@tobi) on the GoToSocial team helped debugging GoToSocial issues (https://github.com/superseriousbusiness/gotosocial/pull/1867, https://github.com/superseriousbusiness/gotosocial/pull/1734, https://github.com/superseriousbusiness/gotosocial/pull/1719),

Daniel (https://mastodon.social/@dansup/) is looking into the work that is needed for Pixelfed; right now Pixelfed incorrectly reports a new notification every time Tusky checks (https://github.com/pixelfed/pixelfed/issues/4461) so if you have a Pixelfed account in Tusky you may want to mute those until Pixelfed is fixed.

Partly a note-to-self, partly a note to anyone else who has the same problem.

Out-of-the-box, Linux (at least 5.15.0-69-generic #76-Ubuntu) does not recognise the fans controlled by the Asus H770-PLUS D4 motherboard.

You can confirm this if running sensors shows no fan RPM information, and if the system gets under any kind of load (e.g., running an Android emulator) the fans spin up to what sounds like maximum speed, and stay there, irrespective of the actual CPU temperatures.

To fix this boot the kernel with the acpi_enforce_resources=lax flag.

On my system (Linux Mint) that was by:

  1. Edit /etc/default/grub, adding the flag to the existing GRUB_CMDLINE_LINUX_DEFAULT entry
  2. Run sudo update-grub
  3. Reboot.

Re-run sensors and confirm that there is fan information.

I occasionally help out with OpenTechSchool Zurich, answering questions from folks getting in to programming. There was a question about the difference between function statements and function variables in Javascript.

This is an edited version of the explanation I gave, in case it's useful to anyone else. It was originally in a chat channel with people reading with a wide mix of experience levels.

“All teaching is the process of lying, and then replacing the lies with successively better approximations of the truth.”

Some of what follows is lies. But it lets us get closer to the truth

The original question was:

Especially what confuses me are the functions as there seems to be existent two type of functions like Function expression AND Function statement.

It's not two different types of functions. It's two different ways of referring to a function.

A quick recap of values, types, and variables

I'm going to recap values, types, and variables, to make sure we're on the same page.

It's important to keep these concepts distinct.

At its core, your program is going to be operating on values. A value might be a number like 3, or a string of characters like “hello”, or many other things.

Each value has a type. It's a bit circular, but this is the type of thing it is. 3's type is a number, "hello"'s type is a string, 3.0's type is a floating point number, and so on.

You can't have a value without it also having a type. They go hand-in-hand together.

Different programming languages have different types available to them. Javascript doesn't have many

Programs would get pretty confusing if we had to use values all the time. Consider the value 3; is it supposed to represent a temperature, a calculation, a shoe size, a distance, ... ?

So we have variables.

A variable is a way of labelling a value with a name.

Think of a variable as being like a box. You put the value in the box, and write a label on the outside of the box telling you something about the value inside the box.

And the box has a window on the side, so you can see the value in the box.

You can think of let temperature = 3.0; as “Create a box, put the value 3.0 in the box, and write 'temperature' on the outside of the box”

Some programming languages require you to say what type of value you're going to put in a variable before you can use the variable.

Javascript does not. You can create a variable containing a string, and then put a number in the same variable later.

let x = "some string";
...
x = 3; // <-- this is allowed

Python is similar to Javascript.

In some other languages, like Java you need to specify the type of value you can store in the variable. Later, if you try and put a value with a different type in the same variable you will get an error.

String x = "some string";
...
x = 3; // <-- this is not allowed in Java,
       // x can only contain values with type "String",
       // and the type of 3 is "int"

Our three key concepts now are:

  • values, like 3, “hello”, and 3.0
  • types, like number, or string. Each value has a type
  • variables, ways of labelling values so they have meaning and can be reused

Javascript can tell you what the type of a value is using the typeof operator.

> console.log(typeof 3);
number

> console.log(typeof 3.0);
number

> console.log(typeof 'hello');
string

> let x = 4;
> console.log(typeof x);
number

The last line is interesting. It will print number, but that's not the type of the variable, it's the type of the value in the variable.

What does this have to do with functions?

Functions are also values, and have types

This is both important, and confusing.

Consider a function like this:

function foo() {
  return 42;
}

What's the type of the function?

Most people will say “number” — the function returns a number, so when they see the question “What's the type of the function?” they actually see “What's the type of the value returned by the function?”.

And the function does return a number. So it's the correct answer to the second question.

To be fair, most of the time, it's what the question actually means, so it's a reasonable assumption.

But it's the wrong answer to the first question. The type of a function in Javascript is function.

You can check this with:

> function foo() { return 42; }
> console.log(typeof foo);
function

ASIDE: Javascript is pretty limited in this respect. Other languages will be more specific, and tell you the type is function () -> number or similar, which means it's a function that takes no arguments and returns a number.

Because it has a type a function is also a value.

For the purposes of this example you can imagine a function's value is all of the code in the function.

If a function is a value, and it has a type, then it's no different to 3, or "hello", or 3.0, which are also values with types (every value has a type!).

This means we can put a function in a variable. Like this.

let my_func = function() {
  return 42;
};

And we can call the function in this variable.

> console.log(my_func);
ƒ () {return 42;}  // <-- output

Wait, what's going on with the output? That's not 42.

Imagine we had done this:

> let x = 3;
> console.log(x);
3

Javascript is taking the variable x, and telling you the value inside the variable. 3 in this case.

That's exactly what's happening with the console.log(my_func); example. The value of the function is being shown to you as ƒ () {return 42;}

If you want to call the value of the function in the variable you have to use () the same as any other function call.

> console.log(my_func()); // <-- note the extra ()
42

And if you try and use a value that isn't a function type you get an error message.

> let x = 3;
> x();
Uncaught TypeError: x is not a function

The error message could be more precise. It should really say “The value in x is not a function”, because in both cases neither x or my_func are functions, they are variables. It's the value inside the variable that's important.

Like I say, it's important to keep the value / type / variable distinction clear.

Function statements and function expressions

Knowing all this we can look at the difference between function statements and function expressions.

A function statement looks like:

function foo() {
  return 42;
}

A function expression looks like:

let foo = function() {
  return 42;
};

These both do the following identical things:

  • Create a variable
  • ... called foo
  • ... that contains a value
  • ... the value is ƒ () {return 42;}
  • ... and the the type is function.

The only difference is where the variable is visible from.

Hopefully you're familiar with variable visibility already. It means that this code won't work as intended.

console.log(x);
let x = 4;

On line 1 you try and use the value in the variable x, but that variable is only created later, on line 2.

It's the same thing with function expressions. If you tried this:

console.log(foo());
let foo = function() { return 42; };

that won't work either. The variable foo is used on line 1 before it's created on line 2.

But this next example does work (in a program, not from the Javascript console).

console.log(foo());

function foo() {
  return 42;
}

When you stop and think about it, this is a bit weird. We're able to use the function in the code before we've defined it. How?

Javascript calls this function hoisting.

A function declared using a function statement is “hoisted up” (for non-native English speakers, “hoist” is a synonym for “lift”) so it's available before any code that uses it.

A function declared using a function expression is not hoisted, so it's available only after it's been declared.

And that's the difference.

Because life's too short to spend it reading terrible interview feedback

This is my rough guide to how to prepare to do good tech interviews and provide useful feedback. It’s not a one-size-fits-all guide; it’s the best practices I’ve adopted after giving more than 400 technical interviews to candidates with all ranges of experience and backgrounds who are applying for roles with a strong software engineering component to them.

It assumes you're an interviewer using the typical tech interview format of a 45-60 minute discussion with a candidate, generally focusing on a specific area of knowledge.

I'm not saying that’s the right way to conduct candidate interviews, or commenting on the various merits of this approach vs. take-home coding interviews, or pair-programming with an interviewer, or any other approach. But if this is the approach your organisation has taken to interviews, this is how you can conduct them well.

Before the interview

I assume:

  • You have time to prepare for the interview
  • You are familiar with the role and level the candidate is interviewing for, and the role’s requirements
  • You are not the final hiring decision maker; you are providing feedback to a hiring manager or committee

Your goals for the interview are:

  • Allow the candidate to perform as well as they can
  • Identify the candidate’s skills and ability at those skills
  • Provide a clear, unambiguous hiring recommendation to the decision makers, comparing the candidates skills and abilities against those required by the role
  • Provide sufficient information about the interview to allow the decision makers to disagree with your recommendation

Set aside time to prepare

You must go into the interview prepared, knowing what you expect to ask the candidate, likely detours, and expectations for their performance.

If you're not used to interviewing allow several hours for this. As you do more interviews this should take less time, but with more than 400 interviews under my belt I still allow at least an hour.

Do this several days before the interview if possible. 30 minutes before the interview is the wrong time to discover you’ve been asked to interview a candidate you’re not qualified to interview.

Form an expectation

You should go into the interview with an expectation as to how the candidate will perform on the questions you ask them. This allows your feedback to focus on whether or not they met those expectations.

This also allows the hiring group to decide your expectations were wrong and evaluate your feedback in the context of their expectations instead.

Your expectations about a discussion on a topic with a candidate with 2 years experience and one with 10 years experience should be very different.

Read the candidate's CV. Pay attention to:

  • Their most recent roles. Anything in the last three years is reasonable to expect them to be good at. Anything before then is going to get increasingly rusty.
  • Specific projects they have worked on, and their role on those projects
  • Specific technologies they have used
  • Anything that overlaps with areas you are an expert in

If they list them, look at any open source or other public contributions the candidate has made. You're looking for things that will let you go into the interview thinking “If I ask the candidate about X then they should be able to do very well”.

Do not form an opinion if there isn't any open source work to review. You have no insight into why, and it is also inappropriate to ask the candidate. This gives you zero signal.

Setting the expectations down before you've met the candidate may also help prevent forms of bias creeping in after you've had the chance to meet them. This is a double-edged sword, of course, be careful of any biases you may have from the information in their CV.

Prepare questions to ask

You probably have a set of questions you're comfortable asking and stick to those where possible. That's ok, but don't blindly repeat the same question interview after interview.

Tailor the topics to the candidate's experience

You want the candidate to do as well as they can. So focus on playing to their strengths, while still being relevant to the role. If they do not meet the expectation you want to be able to say “I gave them every opportunity to demonstrate their abilities”.

For example, if you typically start infrastructure scaling discussions using web services as an example, but the candidate doesn't have experience with web infrastructure, but does with mail (SMTP) infrastructure, adapt your questions accordingly.

Don't: Ask questions you can predict the candidate won't do well on. For example, anything involving the minutiae of a programming language they last used 3 or more years ago doesn't make sense — they're probably too rusty on the specifics to get a useful signal.

Or if the candidate's CV suggests they have experience working with systems of ~ 10 servers, don't ask them to design something that needs ~ 1000 servers — once you go up by more than two or three orders of magnitude the scaling issues and bottlenecks become very different. Since you can already predict the candidate will probably do poorly on the question there's no point in asking it.

Don't: Get hung up on having, e.g., three topics prepared, and you want to make sure you get to discuss all three topics. If you've started on a topic and you've found yourself having an interesting, deep technical conversation with the candidate and you feel like you can continue to explore the topic and learn more about the candidate's abilities then do so.

But, if you feel you have got all the information you need from the discussion (e.g., their answer so far is going to help you make a firm yes/no decision) then you can stop early and move to the next topic.

Start writing your feedback

Once you have your understanding of the candidate, the topics you're going to cover, and your expectations, write those down as the skeleton of your draft feedback.

This helps speed up the process of submitting feedback after the interview.

It also helps guard against any bias happening during the interview. Perhaps the candidate has a particularly charming personality, and you've left the interview convinced they should be hired because of that instead of their skills. Setting out your expectations now helps ensure they are not coloured by experience during the interview.

During the interview

Keep track of time

Pay attention to the time in the interview.

You want to make sure you get through the topics you had planned to cover (but see the previous section, re being flexible).

You should note down how long the candidate spent on certain topics. How long a discussion takes to hit the points you hope to cover should be part of your expectation going into the interview.

Don't Interrupt.

Allow the candidate to think. This is especially important in a design or coding interview.

Silence during an interview while a candidate is thinking can feel like an eternity.

Resist the temptation to fill the void by talking while the candidate is thinking, designing, or coding. Similarly, if you see they've made a mistake do not point it out immediately, wait.

Talking can disrupt their flow of thought — imagine if you were trying to write code with someone looking over your shoulder interrupting every time they noticed a typo.

It also robs the candidate of the opportunity to notice and fix the mistake themselves. Seeing whether a candidate can spot a mistake on their own, and how they fix it, can be useful information.

Make sure you have told the candidate this at the start of the interview, and it's ok for them to ask questions or for help if they get stuck on something. But wait for them to ask, do not offer it unbidden.

After the interview

Finish writing up your feedback.

You need to:

  1. Provide a clear opinion, supported by evidence, as to whether or not you think the candidate should be hired
  2. Provide enough evidence so if the hiring committee decides they disagree with your opinion they have enough information to reconstruct what happened in the interview and form their own opinion.

I recommend you structure your feedback as follows.

  • Summary, consisting of three paragraphs.
    1. Explain what you understood from the candidate's CV
    2. Set out your general expectations
    3. A conclusion, explaining whether you think the candidate should be hired. If it helps, start this with the phrase “We should hire X because...” or “We should not hire X because...” Be extremely clear in explaining your conclusion, calling out specific parts of any answer you think was particularly strong or weak. If you have not come to a conclusion then you've wasted the interview
  • Discussion sections
    • For each topic you discussed set out:
      • The initial question, as you asked it to the candidate
      • Your expectations ahead of the interview
      • Details from the discussions
      • Your assessment of the discussion relative to your expectations

Include code and designs

If the candidate designed something in the interview, include the design. If it went through multiple iterations, include the iterations.

If the candidate wrote code during the interview include the code in your feedback. If it went through multiple iterations, include the iterations.

Be extremely clear in your feedback about what you told the candidate about what is/is not acceptable in code.

For example, you will probably get different code from a candidate if you say “I just want to see something that works, don't worry about making it testable, or checking for errors” than if you say “Please write code you would be comfortable sending for a code review”.

If you told the candidate the former, and the hiring group doesn't know, they may well err on the side of caution and assume the candidate always writes scrappy code.

If the candidate called out anything they said they would normally do, but are omitting for speed during the interview then call that out too. For example, if the candidate is writing a function that has to interact with the current time they might say something like “Ordinarily I'd pass an object in with methods to get the date/time here, because it makes this much easier to test. I'm going to skip that for the moment” then this is valuable (and good!) thinking on the part of the candidate. Note it down.

Appendix

Sample feedback

This is an example of the feedback format, and the level of detail I want to see when I'm trying to figure out if we should hire someone.

If your response to this is “Holy crap, that's a lot of detail, I'm a busy engineer, I don't have time to do this” then I'm here to tell you you're wrong. Growing the size of the team with skilled engineers is one of the largest-impact things you can do. The recruiting effort requires time from coordinators, recruiters, other interviewers, and, of course, the candidate. Don't waste their time by turning in a single paragraph of poorly detailed non-specific feedback.

Summary

Alexa has been working as a developer for 4 years, most recently implementing the design of a backend system for a crowd-funding site, growing 100x in volume over the last two years, written in Ruby. She's part of a team of 6, and appears to have been promoted up the eng. ladder once in those 4 years, which is promising.

Expectations: She should be able to talk about the project in detail and be comfortable writing code. I'm not expecting significant design skills yet due to relative lack of experience and being on a team where others are doing the bulk of the design work.

Conclusion: Hire. She communicates clearly and can talk in a detailed, organised fashion about the work she's been doing. She had sensible ideas about improving the project to scale it out. Her code was clean, and idiomatic, the tests made sense, and she didn't over-engineer things to deal with problems that don't exist. Based on this, I am confident I could give her a project like X and she would be able to complete it independently, which meets the expectations for this role.

Topic: Describe your most recent project

Expectation: She's been on this project for 4 years, 2 as a junior dev, 2 as a mid-level dev, so should have a solid understanding of it, the tradeoffs, and the areas she feels can be improved. She should also be able to explain, in detail, how the system has scaled, and failure modes that have been anticipated and designed around.

What happened:

[Include the narrative here, which I've skipped because I'm writing a blog post, not interview fanfic]

Conclusion: Good.

  • She gave a detailed, organised description of the system, structured around the path a user's request takes (I thought this was a good approach, it introduced each area of the system in turn, and in a relevant context, each explanation building on information from the previous explanation).
  • She understood the current scaling issues the system is facing, and the suggestions she made for fixing them are sensible
  • She was able to talk about how the current system could be split up along service boundaries to re-implement services in more performant languages to deal with future scaling issues.
  • Her proposed approach to testing the migration (modify frontends to send query traffic to old and new backends, compare the results for equality, export data about the result, throw away the result from the new backend) is sensible.

Topic: Write FizzBuzz in your language of choice

The problem is to, for the numbers from 1 up to some limit, print either:

  • “FizzBuzz” if the number is divisible by 3 and 5
  • ... or “Fizz” if the number is divisible by 3
  • ... or “Buzz” if the number is divisible by 5
  • ... or the number

I asked for code she would be comfortable sending for initial code review, but not to worry about documentation comments for functions. I did not specifically ask for tests or any error checking to see what she suggested.

In case it wasn't clear, this is a joke question. Do not ask this. Tom Dalling has a great blog post, “FizzBuzz in too much detail”, https://www.tomdalling.com/blog/software-design/fizzbuzz-in-too-much-detail/ which explores this rabbit hole

Expectation: She lists Ruby as her preferred language, and the project she's been working on for the last four years is Ruby-exclusive, so she should be very familiar with it. I expect initial working code in less than 10 minutes. If she doesn't suggest it, I'll ask for tests.

What happened:

She noted down the problem to refer back to it and repeated it back to me, said she'd focus on getting something that worked first, then clean it up, opened an editor, and in just over 3 minutes had written:

def fizzbuzz(limit)
  1.upto(limit) do |i|
    if i % 3 == 0 && i % 5 == 0
      puts 'FizzBuzz'
    elsif i % 3 == 0
      puts 'Fizz'
    elsif i % 3 == 0
      puts 'Buzz'
    else
      puts i
    end
  end
end

This has a bug (she repeated the i % 3 case for Buzz). Before running the code she re-read it through and spotted the problem and corrected it without my prompting.

After fixing the code she ran it and confirmed it worked. Unprompted, she said she'd like to quickly re-write it to make this problem less likely. I agreed, and she re-wrote it to:

def fizzbuzz(limit)
  fizz = (i % 3 == 0)
  buzz = (i % 5 == 0)
  1.upto(limit) do |i|
    puts case
      when fizz && buzz then 'FizzBuzz'
      when fizz then 'Fizz'
      when buzz then 'Buzz'
      else i
    end
  end
end

She wondered aloud about whether this should accept more parameters to specify the text to print, or the specific divisors to use, and decided this was probably over-engineering, unless there was a business case for it. I agreed.

She said she'd want unit tests for it before it was ready for review, and I agreed. She started sketching out a test, then quickly realised the function has side effects (printing the output), so rewrote it to return the results, and a set of tests to verify the return value.

# Assume I'm sufficiently familiar with Ruby to write that code and test here.
# I'm sure you get the gist

This was the final code. We spent 25 minutes on this topic.

Conclusion:

Good:

  • Most importantly: Wrote working code that solved the problem
  • Wrote the problem down first, repeated it back to confirm understanding
  • Focused on getting something working instead of over-optimising from the beginning
  • Read the code to double check before running it, spotted a problem, fixed it
  • Implemented sensible refactorings
  • Suggested writing tests without prompting, as part of making it review-ready
  • Good sense to not over-engineer code without reason

Could be better

  • Could maybe have realised earlier that printing the results directly would make the code more difficult to test. Although as an iterative approach to development I'm fine with it.

I'm worried by a trend I see where software intended for use in production systems is defaulting to automatically loading configuration information from environment variables.

I think this is a bad idea.

Initial configuration to a production system should be explicit and visible, so configuration should come from one of two places:

  • Command line flags
  • Configuration sources referenced by command line flags (e.g., URL to a service that the process can contact to fetch additional configuration)

Note: How those command line flags are generated and provided is a separate, orthogonal, story. Maybe it's a systemd service file. Or a Kubernetes job definition. Or an Ansible role. Or something else.

Using environment variables by default is an attractive nuisance that complicates troubleshooting and reproducibility.

Troubleshooting

When troubleshooting a service I'm almost certainly going to need to try and understand what its configuration is.

The service might provide a dedicated interface to see this, but since there's no standard for this each service's mechanism is likely subtly different. This more information to have to recall when managing a system.

Everything accepts command line flags. So looking at the command line a program was started with (whether that's in a log line, from running ps(1), inspecting the service file, reading the Kubernetes configuration, on a dashboard somewhere, ...) is the universal way to do this.

Using environment variables by default breaks this. Now I have to have a mechanism for seeing the environment variables the service was started with. Either this is going to show me all the environment variables, and I have to remember which one was relevant. Or the mechanism is going to have to know, on a service by service basis, which ones are important and only show me those.

This, and any other mitigation measures like this are added complexity that could be avoided by not automatically using environment variables in the first place.

Reproducibility

This is closely linked to troubleshooting, because when troubleshooting a problem it helps to be able to reproduce it.

If your service is reading configuration from environment variables it no longer suffices for someone to file an issue and provide a simple command line to reproduce the problem.

Now they need to know that the service is automatically loading environment variables, and they need to provide those as well. And if they don't you then you have to (a) know that these variables are important, and (b) get back to them and ask for this additional information.

When is it OK?

There are some occasions where this is OK.

It's not a production service

For non-production services, sure. Anything goes. As it says at the top of this post, these are my rules for handling production services.

It's an extremely common environment variable

There are a number of environment variables that are so widely used that it's impossible to escape using them. Things like HOME, LC_*, PAGER, TZ, and so on.

https://pubs.opengroup.org/onlinepubs/7908799/xbd/envvar.html has a good list.

Even if you do this you're still creating troubleshooting and reproducibility problems for yourself, especially with the LC_* and TZ variables.

In fact, I think there's an argument to be made that production-grade software that deals with locale related information should refuse to read the LC_* and TZ variables and require that that information be provided as explicit configuration.

If that means that the command line is a bit longer because it has a --tz=${TZ} entry I think that's a small price to pay for making critical configuration information like this completely explicit.