HTTP-P2P, HTTP with more Ps

Marco Munizaga
HTTP-P2P, HTTP with more Ps
#http #go
Share this item:

# Introduction

We're introducing a new experimental API (opens new window) in go-libp2p, enabling developers to utilize libp2p with the well-known semantics of HTTP. This isn't a special flavor of HTTP; it's standard HTTP, but enhanced with libp2p. Developers can now benefit from HTTP intermediaries such as CDN caching (opens new window) and layer 7 load balancing (opens new window). This allows developers to create HTTP applications that operate over NATs and seamlessly tap into libp2p's diverse transport options to boost connectivity. In addition, the HTTP transport now joins the roster of supported transports in libp2p.

Here are some use cases we are excited about:

  • A peer can fetch content from another peer over the IPFS Path Gateway (opens new window) protocol, and it will work regardless if the remote peer is:
  • HTTP Edge compute nodes can now behave as peers in the libp2p network.
    • Many edge compute nodes are constrained to either HTTP or WebSockets, with a premium cost on WebSockets.
  • Simple HTTP clients like curl can now participate in the libp2p network.
  • A browser can make a secure HTTP request to a peer using WebTransport or WebRTC, and thus avoid having to rely on Web PKI.
  • Operators of bigger libp2p deployments can use layer 7 load balancing to route and scale their protocols easily. Projects like Envoy (opens new window) are now libp2p compatible.
  • Throw away your long running virtual machine. Have peer browsers act as HTTP servers for each other. Keep state as a CRDT that is maintained by every peer of the application (Ă  la gossippad (opens new window)).
  • Port existing HTTP applications to a p2p environment.

We’re hoping to get some early feedback as the API solidifies, so please try it out and let us know what you think in the go-libp2p Discussions forum (opens new window).

# Technical details

This new api is an implementation of the libp2p+HTTP spec (opens new window). The main features are:

  1. Defining a new HTTP transport
  2. Providing HTTP semantics to users that work on any transport.
  3. Defining a .well-known/libp2p resource for learning about a peer.

# A new HTTP Transport

A libp2p node can now listen on an HTTP transport and advertise its address as a multiaddr ending in /tls/http (or, equivalently, /https). For example, a libp2p node that is listening on port 443 would have a multiaddr of /ip4/1.2.3.4/tcp/443/tls/http. If the node has a domain name it could use that as well, i.e. /dns/example.com/https. The HTTP transport lives alongside the node’s other transports (tcp+tls+yamux, QUIC, WebRTC, etc). The key difference is that the HTTP transport only supports HTTP requests and responses, and thus does not support the stream-based interface that other libp2p transports support.

The HTTP transport is a normal HTTPS server/client. There’s is nothing libp2p specific about it. This is on purpose as it allows us to interoperate with the wide existing HTTP ecosystem.

# HTTP Semantics

HTTP semantics differ from the HTTP transport in that they are the abstract form of HTTP. They don’t specify how an HTTP message will be encoded and sent to a remote node, they only specify what an HTTP message is and its interpretation. These semantics are defined by RFC 9110 (opens new window). This difference means we can adopt HTTP semantics without limiting ourselves to only an HTTP transport. We can use WebTransport, WebRTC, or a hole-punched QUIC connection to make an HTTP request. This allows developers to create applications using familiar HTTP tools in a p2p setting. On a technical level, this is implemented by opening a new stream for every HTTP request and encoding the HTTP message as HTTP/1.1.

# .well-known/libp2p

The last feature solves a problem that’s unique to running HTTP in a p2p setting. It’s about protocol discovery and signaling. Given that you know about a peer, how do you learn about the application protocols they provide? Do they provide an IPFS Gateway (opens new window)? Do they index CIDs and can be queried over the IPNI interface (opens new window)?

In a traditional HTTP setting, servers are well known, and you have out-of-band knowledge about what they support. api.foo.example.com is the API for the foo service on example.com. But in a p2p setting, you may only have a peer’s Multiaddr (e.g. /ip4/127.0.0.1/udp/49926/quic-v1/p2p/12D3KooWExdwiYFTpSbvtvpPvig3X8u2PLbd7QyNJHqpGzHYd8Dq). You need a standard way to ask this peer about the features it supports. That’s where /.well-known/libp2p comes in. A node provides information about what protocols it supports and where they are mounted at in the .well-known/libp2p resource. For example, a node that supports the /hello/1 application protocol and mounts it at /hello-path/ would return the following .well-known/libp2p (JSON encoded):

{"/hello/1":{"path":"/hello-path/"}}

This feature is optional. If you have some out-of-band mechanism for signaling supported protocols, you can use that instead of .well-known/libp2p. In the future, we expect this information to reside alongside a peer’s address information because it’s useful to know not only how to communicate to a peer, but also what protocols they support. For example, we may want to know if a peer supports the IPFS Gateway protocol before going through the effort of connecting to them. In this future, .well-known/libp2p will still provide the most up to date information about this peer, and can act as a graceful fallback.

Readers familiar with libp2p may recognize a similarity to the identify protocol that is run over streams. These solve a similar problem. The reason to introduce this new thing is to fit better with HTTP conventions and semantics, as well as to trim things we don’t need.

# How to use this

Right now, the best support is in go-libp2p and there’s some preliminary (albeit rough) support for js-libp2p with js-libp2p-fetch (opens new window). We hope to see first-class support across implementations, but until then you can still use this, just with a bit more effort. See the Other environments section below.

# In go-libp2p

go-libp2p has had the most time spent in crafting first-class support for libp2p+HTTP. The core idea is that you create an HTTPHost that is similar to a stream based libp2p host.Host except it’s built around HTTP semantics. The HTTPHost has methods to .Serve HTTP handlers over various transports, and make http.RoundTripper s or http.Clients for specific peers. Refer to the examples and the documentation in the godoc for more information. And file an issue (or a small PR) if any examples or documentation can be improved.

# In js-libp2p

The browser is a platform where libp2p+HTTP can truly shine. Build your application using the standard [fetch](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) API and then use either the native browser fetch implementation for an HTTP transport (the remote peer’s multiaddr ends in /tls/http or /https) or js-libp2p fetch (opens new window) for doing HTTP over a libp2p stream transport.

There’s still some manual switching of the underlying fetch implementation to use, but we hope that will become easier to use in the future. API suggestions and ideas here are welcome.

# In other environments

A goal with this work is to be interoperable with existing HTTP systems. Or, put another way, it is to make all existing HTTP systems libp2p-compatible. That means you’ll be able to interact with a go-libp2p node with curl, or get rid of your long running virtual machine and replace it with ephemeral on-demand edge compute. You’ll be able to create your p2p applications on top of HTTP semantics, and then leverage libp2p to run it everywhere.

If you’re interested in how to add libp2p+HTTP support to your environment, take a look at the HTTP Spec (opens new window). It’s relatively small, and can be added on top of an existing libp2p-implementation (no core changes required).

# Prior Art

This isn’t the first time folks have wanted to give HTTP p2p super powers. Here is some prior art in this field:

  • go-libp2p-http (opens new window), started in 2017, encoded HTTP messages as HTTP/1.1 and sent them over a libp2p stream with the default protocol ID of "/libp2p-http". The new libp2p+HTTP API follows this implementation closely and is compatible with it, but standardizes on the protocol ID of /http/1.1 to signal that this is an HTTP/1.1 encoded HTTP message.
  • Kubo (formerly known as go-ipfs) in 2019 used the above go-libp2p-http package to proxy HTTP requests over libp2p streams with an experimental feature called p2p-http-proxy (opens new window).
    • This feature is used by Peergos to run their HTTP applications in a p2p way. More info at https://peergos.org/posts/dev-update.

This new work builds upon past work and tries to standardize how HTTP over libp2p streams works with the libp2p+HTTP spec (opens new window). This work also embraces HTTP transports as a way (but not the only way) to send HTTP messages.

# Future extensions

  • Support libp2p Peer ID Authentication over HTTP: https://github.com/libp2p/specs/pull/564
  • Support using generic request/response semantics rather than HTTP semantics: https://github.com/libp2p/specs/pull/561