RESTful API design at c.technology

During the development of our API version 2.0, we spent a great deal of time gathering as many best practices and commonly used conventions on RESTful API design as possible. Like many others, we agreed on using OpenAPI early on, but this (and the choice to build a REST API - with all its caveats and "left-to-the-implementer" parts) still left a lot of design choices up to us.

In this post, we highlight our API design choices, starting with rather widespread ones, ending with more debateable ones, and discuss why we do it the way we do it. This post serves as an internal guideline, but might hopefully also help others in the same situation.

Base URL of the API

A very basic choice concerns the use of a subdomain vs. a path prefix:

• api.ctechnology.io/v2.0/...: Makes it very easy (and on the level of the DNS) to separate the API service from the (likely more customer-facing) website (and potential other services running under the same domain, such as docs.ctechnology.io or app.ctechnology.io). However, this also requires implementing CORS when accessing the API from a web application, thus making everything slightly more complex.
• ctechnology.io/api/v2.0/...: Potentially requires more elaborate routing to separate the API from other components of the system, but makes it very straightforward to bundle a complete application, e.g., by using a web framework that both offers website creation as well as API provision and data management.

Like many, we started out using an "all-in-one" web framework and thus hosted the API under the ctechnology.io domain. Once we started splitting of the c.technology website into its own service (with an improved CMS at its base), we started transitioning towards hosting the API under its own subdomain (api.ctechnology.io) in order to reduce the routing complexity and improve the system resilience.

Versioning and compatibility

For the versioning scheme we decided to use SemVer early on, and were primarily discussing if the patch version should be included or not, and if we should prefix the version with a v or not. Taking inspiration from other, popular APIs, we saw that it is rather uncommon to include the patch version in the URL (a detailed discussion why follows below), and the “version”-v is not uncommon, though there seems no strict reason as to why people add it (except hinting at the following numbers being a version identifier).

We have the following rules in accordance with SemVer (where the API versioning is defined as major.minor.patch):

• patch version bumps keep the same API, resp. the API as defined in our OpenAPI specification*. Thus, the patch version does not have to be specified when using the API.
• minor version bumps may add fields and parameters (both to queries as well as responses), but not change or remove existing ones.
• major version bumps may remove, rename, change fields, parameters, endpoints - basically everything that is in our API.

If you use a library that gracefully handles additional fields in endpoints, you may use v2.x as API version specifier, and we'll always use the newest available minor version. You are doing this at your own risk, though!

Considering all these aspects, we ultimately opted for api.ctechnology.io/v2.0/... as it allows us to completely (and easily) separate the API from the ctechnology.io website (and other services under the same domain), and as only requiring minor versions means users do not have to change their patch versions to get the newest bugfixes, and we do not have to create different endpoints for API-compatible changes.

*If the implemention is wrong, bugfixes that make it match the specification are allowed - so if you encounter any discrepancies, let us know quickly, and we'll fix it in no time.

Resources, paths, request methods

This part of an API is rather well-specified within REST or at least agreed upon. We differentiate between single resources and resource collections.

Single resources

A single resource is always made available at an endpoint that ends in the resource's ID, except if there is a 1:1 mapping to another resource that is specified at another part in the path.

For example, a vehicle (profile) is accessible under /vehicle/{id}, but the vehicle's most up-to-date state might be accessible under /vehicle/{id}/status. We chose this approach over /vehicle/status/{id} to

1. make it clear that you have to use the vehicle's ID to access its status (and not the "status resource"'s ID - that ID is hidden from the API user in any case) and to
2. allow easier prefixing (most things that have to do with a particular vehicle are available under /vehicle/{id}/..., and thus certain prefix-based caching libraries may be used more conveniently).

For single resources, we use the following HTTP methods:

• GET: Returns a single resource, by default as JSON object (but respecting the Accept and Content-Type headers).
• PUT: Accepts a partial representation of the resource, and returns the updated resource (by default as JSON object - exactly the same resource as the GET request would have yielded).
• DELETE: Returns an empty HTTP response with status code 204.

Note that we do not use PATCH at all - in our opinion it is simply not common enough, and usually used interchangeably with PUT (of course, there are semantic differences, but we find that they don't matter enough in practice to warrant handling of two different verbs).

Resource Collections

Resource collections can (and must) be accessed using endpoints ending in a slash /, for example, /vehicle/ will return all vehicles that a user has access to. As you might have already realized, most commonly the endpoints are exactly the same as when accessing a single resource, just without the resource ID.

• GET: Returns a collection of resources, by default as a list of JSON objects. An individual resource has the same type / shape as the resource retrieved via /resource/{resource_id}.
• POST: Accepts a complete representation (all fields specified as non-optional and not read-only) of a resource, and creates this resource. A successful request will return the newly created resource (where its assigned ID and potentially computed attributes can also be found). We do not return the resource collection upon a successful POST request as the corresponding GET requests often have additional parameters (offsets, limits, filters, etc.), and thus the semantics of the returned collection would not be well-defined enough (consider, e.g., that the newly created resource would seldom be part of the collection returned "by default", and thus it would become very difficult to track down it's ID, etc.).

All resource collection GET requests support pagination "out of the box". The pagination information is part of the envelope header (see below), and essentially consists of offset, limit and num_results information (the number of results might not be given in case it would be exhaustive to compute it; in this case num_results is the "magic number" -1).

Side effects

Occasionally, it is required to trigger something in the backend that isn't strictly related to a resource, or we need some ephemeral data that has no resource associated. Out of convenience (and to allow such "remote procedure calling"), we specify the following two path elements that help understanding when a path isn't tied to a resource:

• .../ops/...: Triggers an operation on the backend that does not result in the creation of a resource nor is bound to any. For example, /vehicle/{id}/ops/notify-users could trigger a job that sends notifications to users who have access to the vehicle.
• .../ask/...: Creates an "ephemeral resource" in the sense that the backend simply computes something and returns it, without creating an associated resource. For example, vehicle-service/{id}/ask/projected-fulfillment-date could return a projection when a certain service could be completed.

Filtering

Many resource collection GET requests allow filtering and / or field selection. In the following we discuss the most important filtering mechanisms.

• General filter (?status=...): Filters the indicated field according to the given text (or primitive value, such as a number or a boolean value). The possible filter values are given in the OpenAPI specification. For selected numerical fields, __lt (less than), __lte (less than or equal), etc., might be available, and for certain fields __not might be valid (logical negation).
• Temporal fields (?timestamp__gt=...&timestamp__lt=..., ?timestamp__date__gt=...&timestamp__date__lt=..., ?timestamp__time__gt=...&timestamp__time__lt=...): These fields may additionally accept inputs in date or time formats (in addition to our default (ISO 8601 conform) datetime format YYYY-mm-ddTHH:MM:SS.fZ), and filtering on these more granular specifiers.
• Filtering returned fields (?fields=a,b,c): Especially raw data retrieval endpoints may accept a fields query parameter that lets an API consumer specify which fields to retrieve from a resource (or collection). This is mostly for performance reasons.
• Pagination filtering (?after_obj=...&before_obj=...): In addition to "regular" pagination filtering (with offset and limit), certain endpoints also allow specifying "after" or "before" which object the next page should be.

Envelopes

Using HTTP headers vs. body / JSON envelopes is cause for heated debate. Formally, using HTTP headers such as Pagination-Count, -Page or -Limit is the correct way (and we might add these headers in the future), but from a usability point of view we found it much more convenient to wrap all responses in an envelope, where we define our own (application-specific) headers, such as num_results, offset or limit. This is primarily because

1. auto-generating libraries from our OpenAPI specification makes interaction with these custom headers a breeze,
2. they are easily extensible without having to rely on custom HTTP headers (e.g., adding a custom status code, a translated error message, or a corresponding WebSocket channel on which a user can subscribe to updates),
3. they are easily read- and usable by any user (without having to inspect and / or create custom HTTP headers, which is always an additional step when using any HTTP request library), and
4. they allow us to use the exact same data structures in our AsyncAPI (that does not come with any headers "out of the box", but always requires applications to define their own metadata envelopes and formats).

In short, we always use the following structure:

{
  "header": { ... },
  "data": [ ... ] or { ... }
}

If an API consumer is not interested in the header, this can always and very straightforwardly be circumvented by accessing (only) the data field.

Note that we do not discuss linking to other resources in this article, as we found it is often not required or used in practice. Consider reading about HATEOAS to get more information.

Naming conventions, IDs

In general, we use kebab-case for anything that directly appears in the URL before the query parameters. This seems to be the most widely used / best practice for REST APIs (and URLs in general). For anything else (query parameters, JSON objects passed back and forth, etc.), we use snake_case, but this is simply by convention, as we use this style for many of our backend microservices (and we prefer forcing the API consumers / frontend frameworks to convert into their preferred style, as anything else would preemptively assume a certain style for frontends).

All IDs used throughout the API are passed as strings (even if some might actually be numbers). This may seem slightly more inefficient, but is much more flexible and aligns with our future goal of only handing out intractable IDs to API consumers (that prevent inferring anything about the overall system usage, e.g., by leaking the number of vehicles an account has access to).

Wrap-up

That's it in essence! You can find our API in the "REST API" section at docs.ctechnology.io. If you have any inputs, suggestions, or reasons why you (would) do something differently, feel free to reach out to us via Twitter!

We will publish a similar article where we talk about our AsyncAPI design choices soon - as you might have guessed, many of the choices for our REST API were made by carefully considering what also works well with an AsyncAPI (where there is much less consensus on best practices for API building).

If you are interested, we might also follow-up on this article by talking about our API road- and release map, the advancement and deprecation strategy, library and documentation generation, integration with API discovery services, and more.

Insightful resources

The following might be of use to you for further reading:

• Best practices for REST API design (StackOverflow)
• RESTful API Design Tips from Experience
• When in my REST API should I use an envelope? If I use it in one place, should I always use it?
• Use of PUT vs PATCH methods in REST API real life scenarios