When dealing with large amounts of data, it is important for APIs to offer the ability to retrieve the results that are queried in chunks (aka pages). This allows the requester to receive a timely answer that they can preview while also making sure the request doesn’t time out.
Content management software is no exception to the rule. As such, we’ll review the different options that AODocs offer for pagination in its API.
The different kinds of pagination
Offset-based pagination
The first – and oldest – way to paginate results relies on an offset and a page size. The offset represents the number of results that should be skipped while the page size indicates how many results to return.
A client can start by requesting the first page of 100 results (e.g., offset of 0 and page size of 100), and then decide to either get the next page (e.g., offset of 100 and page size of 100) or maybe request the last page. Assuming 987 results, a client would use an offset of 900 and a page size of anything over 87. However, we would usually stick to 100.
This method has the advantage of being straightforward and is really useful if we want to allow the requester to arbitrarily jump to any page. Notably, browsing all results tends to be more error prone because results being added or removed between two calls causes the offset to be unreliable and less performant.
It’s also worth noting that, depending on how this is implemented, fetching a page with a big offset may take a very long time to return. Some AODocs APIs perform progressively worse the higher the offset.
Cursor-based pagination
Cursor-based pagination relies on a “cursor” which points to a specific item. As such, we can have a named cursor that the backend stores and updates as we advance through the results. Such a cursor can be called by name (as is often used in databases). In the case of a stateless API, each page of query results can include a token; that token can be used in the next call to continue from where we left off.
When using this approach, it is not possible to jump to a specific page, but overall the cursor performs better and is more reliable, especially when processing large result sets. Examples include infinite scrolling that always requests a page at a time or when reading the whole resultset. In these cases you might as well request it sequentially.
What about AODocs’ REST API?
All four options (i.e., none, offset-based, cursor-based, or both) are available across AODocs’ REST API. We’ll go into more details for each of those scenarios.
None
A lot of the calls in here with no pagination options are meant for listing items that are very unlikely to be too numerous. For example, listing the permissions on a library (/library/v1/{libraryId}/permission) does not offer the ability to paginate. If you have a need for pagination, it might be a good idea to start making a bigger use of groups.
Some of the calls are more debatable, like a lot of the relation API where the calls to list all relations (/relation/v1/document/{documentId}, /relation/v1beta1/document/{documentId}/relation/{relationId}), or calls to list candidates for relations (/relation/v1/relations/{relationId}/side/{side}/documents, /relation/v1/relations/{relationId}/side/{side}/documents/{documentId}) do not offer any type of pagination.
Others allow a way to limit the amount of results but no offset or page tokens. This is used when we don’t have a need to get anything past the limit given. This is the case for /library/v2beta1/suggest.
Some calls, however, do not allow pagination while it could definitely be helpful. This one is a bit disappointing as we are left with no workaround in some scenarios.
In all of those cases, the data fetched is hierarchical and AODocs probably hopes that you fetch it one level at a time, like categories, (/category/v1/libraries/{libraryId}/categories/{categoryId}/allvalueshierarchy), folders, (/folder/v1/files/{folderId}/children), and users (/user/v1/group/{groupEmail}/members).
If any of these has a lot of children under the same level, it can get troublesome.
Offset-based
AODocs has a few calls that only support offset-based pagination. The parameter used for the offset is simply called “offset” in /document/v1/{documentId}/version and /document/v1/drive/{driveId}/version, while it is called “nextPageOffset” in /folder/v1/libraries/{libraryId}/foldersAndDocuments.
Unless you have a use case where your documents have thousands of versions, and you might browse all the way through them rather than only view the most recent/oldest ones, the first 2 calls should not be something where the limitations of the offset-based approach hurts you. The third call is used to retrieve deleted documents. It might be an issue if you have a lot of deleted documents and have a use case where you need to browse through them all.
Cursor-based
This is by far the most common pagination option used in AODocs. The Google Drive API itself makes extensive use of those tokens where the query parameter to pass a token is called “pageToken” while the token returned by the API will be a field at the root level called “nextPageToken” and the number of results to return is defined via a parameter called “pageSize”.
AODocs is a bit less consistent with the name of its field. The parameter to pass the token in the query is almost always called pageToken, but a couple of calls have it as “cursor”. Some calls have both “pageToken” and “nextPageToken” but the latter is marked as deprecated, so “pageToken” will be the one used across the vast majority of AODocs API.
The field in the response that contains the next page token is a bit less consistent. The vast majority is using “nextPageToken” like the Drive API, but the same couple of calls from before use “cursor” as that field, and several calls use “pageToken”.
Finally, the number of items to return is mostly called “limit”, which differs from the Drive API choice. But a few calls use the same “pageSize” name that the Drive API uses.
Both
Some calls support both the offset-based and the cursor-based approach, despite it not being clear in the available query parameters.
This comes with a heavy disclaimer as it is not documented in AODocs’ official documentation. While we’ve asked Altirnao to officially support the approach, we have not heard back. As such, it is possible that future versions of AODocs will stop supporting this approach without any warning.
Those calls are both in the Search API, which AODocs is planning to fully rework in 2023. It might become a moot point in the future, but it is currently a very useful trick.
The two calls in question are /search/v1/libraries/{libraryId}/search and /search/v1/libraries/{libraryId}/views/{viewId}.
While those two calls appear to be simple offset-based calls with only the “offset” and “limit” parameters, they still return a page token called “nextPageToken”. It turns out that we can use this token in the “offset” field and use that call as a cursor-based pagination instead of using a number in the offset field.
This is extremely useful as there are a lot of use cases where you might want to run a query that has thousands of results and page through all of them to log something or perform an action on them. In one of our use cases, we had to run through all 7000+ results for a search and it was timing out after running for 10 minutes. Switching from offset-based pagination to cursor-based pagination made that job complete successfully in less than 5 minutes. When using the offset approach, the higher the offset is the higher the delay. We’ve seen it take as long as 30 seconds to retrieve a single page of 100 items when the offset was around 9000. Fetching that same page with the proper page token took no longer than retrieving any other page.
As a side note, it is worth knowing that those two calls have another hidden caveat. While the “limit” attribute only states that it has a default of 20, it also has a cap at 100 results returned. Making this API call with a limit set at 150 will have the call succeed but only return 100 documents per request.
Conclusion
AODocs API is very convenient and allows Zia Consulting to create all sorts of extensions on top of what’s already available in AODocs itself. Not knowing its ins and outs may result in unpleasant surprises while you go live and the amount of content your extension is working on increases drastically. Knowing what to paginate and how can save you from request time outs and make massive performance improvements. Reach out to us today and we’ll make this process easier and more efficient for you.