Quick notes on Mastodon pagination

Prompted by the questions in https://hhmx.de/@nick/16376, the answer's too long to fit in a post.

Doing some monitoring how a original #Mastodon server responds, i never (!) saw the described link header! The response may include no link header or a header like this: Link: ; rel=“next”, ; rel=“prev”

You should get a Link header from any API response that returns a page of data, like https://docs.joinmastodon.org/methods/notifications/#get.

[Almost; in testing this I just discovered an edge case, which I've reported as a bug at https://github.com/mastodon/mastodon/issues/25495]

An API response for a single item, like https://docs.joinmastodon.org/methods/notifications/#get-one won't include the Link header.

I just tested this with https://restfox.dev against my mastodon.social account (https://github.com/tuskyapp/Tusky/pull/3610/files?short_path=00c65c2#diff-00c65c2349f395f9f9b78b0fe84d3638b78f3ef4abf98294364b5c4d0ef32473 has details on how to do this).

On my research i saw something in the #Tusky source, that looks like Tusky as clients seems to generate a link header? If so, why?

https://github.com/tuskyapp/Tusky/blob/develop/app/src/main/java/com/keylesspalace/tusky/components/notifications/NotificationsPagingSource.kt#L107-L178

That function needs to return a page of results as an HTTP Response. The code that calls that function unpacks the Response and parses the contents of the Link header, so the header must be present.

To figure out what page of results to return one of the parameters to getInitialPage is the ID of the item (in this case, a notification) that should be at the start of the page. This is the params.key value.

The Mastodon API lets you make requests like “Get me the page immediately after ID X” or “Get me the page immediately before ID X”.

But it does not let you make a request “Get me the page that includes key X”.

So the code has to do that itself, starting around line 134. It does this by making two API calls. One to retrieve the notification with ID X, and one to retrieve the notifications immediately after that.

If those calls succeed the code has two values to operate on; the notification, and the page of notifications immediately after it.

Remember that this function needs to return a single page of notifications. So it creates a fake page which contains both values. This fake page needs a Link header (because the code that calls this code expects a Link header).

So that's why it constructs one in this case.

This code is only called to get the first (or initial) page to show the user. After that the pages above and below are loaded with single calls to the Mastodon API, and return the Link header without any alterations.

Besides of details, if the header should be named “Link:” or “link:“,

HTTP headers are case-insensitive (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers) so you can use either.

Tusky 22 now sometimes reports an array index out of range (-1) or similar (taking this from my memory, for exact message, i have to check it again).

That would be a bug, a detailed report with reproduction steps would be appreciated.

During development i also saw Tusky sending requests for notifications in an endless loop... wondered why.

I suspect the min_id / max_id handling in your server is not correct.

And... could someone clarify the difference between sinceid and minid? If Mastodon will handle this differently?

sinceid = String. Return results newer than this ID minid = String. Return results immediately newer than this ID

What should be the difference between “newer” and “immediately newer”?

Suppose the user has 10 notifications, numbered 1 through 10.

  1 2 3 4 5 6 7 8 9 10
  ^                 ^
  |                 |
oldest            newest

They make the request:

/api/v1/notifications?since_id=5&limit=2

This means they want two (limit=2) notifications that are newer than notification 5 (since_id=5).

The server returns notifications 9 and 10, as these are the two newest notifications.

Think of since_id as “Start at the newest notification, count backwards limit notifications, and return from there”.

Same setup, but the request is now:

/api/v1/notifications?min_id=5&limit=2

This means they want two (limit=2) notifications that are immediately newer than notification 5 (min_id=5).

The server returns notifications 6 and 7, as these are the two notifications immediately newer than 5.

Think of min_id as “Start at min_id, then return the next limit notifications”.

Given that min_id was added to the API after since_id was my working theory is that this was a design flaw in the API, which min_id fixed after the API was first released.