Access to remote peer's TLS certificate

[ThomasLamprecht]

Hi! We have a use case where our users can pin certificates via their fingerprint, with ureq 2 this was really simple as one could pass a full rustls ClientConfig to ureq using the AgentBuilder's tls_config function and wire that up to use a custom verifier through calling with_custom_certificate_verifier on that rustls ClientConfig builder.

With ureq3 the TlsConfig is a ureq specific type that does not expose setting up anything like that, at least not FWICT.

Am I right with my understanding that we would need to impl our own Connector, probably copying most of the impl<In: Transport> Connector<In> for RustlsConnector to be able to manage the same in ureq 3? Mostly asking to ensure I'm not overlooking something obvious.

And as fingerprint validation is a very simple way for some environments to ensure a safe and trusted TLS connection without the need for any cert store/CA, would you be open for making this a bit simpler to set up–if I did not overlook something existing already.

[ThomasLamprecht]

I rechecked other issues and found this comment, which also mentions creating their own transport/connector to be able to configure more details:
#952 (comment)

FWIW, their linked code also has a fingerprint based verifier.

[algesten]

This question from this week might also be relevant: #1084

[algesten]

I'm investigating the fingerprint checking specifically to decide if that's something we want to have in ureq.

[ThomasLamprecht]

Ah, thanks for that pointer (talk about me overlooking something obvious...) and thanks for your consideration!

[algesten]

I've been looking into accessing the remote peer's certificate to allow for some kind of bespoke verification (such as comparing fingerprints).

To make this a standard function in ureq, it needs to work across the two TLS backends we support: rustls and native-tls. There seems to be two ways of going about it, but only one that both backends support. For rustls, we could implement a bespoke verification as part of the ClientConfig. In native-tls that doesn't seem to be possible.

However, in both, there is a way to access the remote peer's certificate once the handshake is over.

• https://docs.rs/native-tls/0.2.14/native_tls/struct.TlsStream.html#method.peer_certificate
• https://docs.rs/rustls/0.23.28/rustls/client/struct.ClientConnection.html#method.peer_certificates

However herein lies a catch 22, both the rustls and native-tls connectors are lazy. They do not start any handshake until we start some IO (such as writing the request + request body). And these peer certificates are not available until after the handshake.

It seems wrong to wait until after writing the request – if we are connected to an adverse server that will be rejected by fingerprint validation, there might be secrets in the request that we don't want to give away.

Can we force the handshake to happen without reading or writing?

Maybe. I tried calling flush(), and that seems to work. However I don't know if this is guaranteed to work always on all platforms.

Another idea is to set_nonblocking on the underlying TcpStream do a read, and expect ErrorKind::WouldBlock. That is probably more reliable (this is similar to code we already have for probing freshness of pooled connections).

[mcr]

To make this a standard function in ureq, it needs to work across the two TLS backends we support: rustls and native-tls. There seems to be two ways of going about it, but only one that both backends support. For rustls, we could implement a bespoke verification as part of the ClientConfig. In native-tls that doesn't seem to be possible.

I'm unaware of a TLS stack that doesn't have a callback, but I agree that it might not be passed up all the time.

However, in both, there is a way to access the remote peer's certificate once the handshake is over.

• https://docs.rs/native-tls/0.2.14/native_tls/struct.TlsStream.html#method.peer_certificate
• https://docs.rs/rustls/0.23.28/rustls/client/struct.ClientConnection.html#method.peer_certificates

However herein lies a catch 22, both the rustls and native-tls connectors are lazy. They do not start any handshake until we start some IO (such as writing the request + request body). And these peer certificates are not available until after the handshake.

I suspected this might be the case. In my tests, I see the socket open, but I can't tell if the TLS handshake has occurred yet.

It seems wrong to wait until after writing the request – if we are connected to an adverse server that will be rejected by fingerprint validation, there might be secrets in the request that we don't want to give away.

Yes. In my case, my first request needs to include the peer certificate in a (COSE/JOSE) signed request.

Can we force the handshake to happen without reading or writing?

Maybe. I tried calling flush(), and that seems to work. However I don't know if this is guaranteed to work always on all platforms.

Where/how did you call flush?

Another idea is to set_nonblocking on the underlying TcpStream do a read, and expect ErrorKind::WouldBlock. That is probably more reliable (this is similar to code we already have for probing freshness of pooled connections).

I'm not sure I understand how that would kick the TLS negotiation.

I'm not clear how I'd get to the underlying TLS stack's peer_certificate: I think that I have to write code for ureq to do that.
I'm also looking at the rustls config cache, and I don't want any of that: I want to provide the Rustls ClientConfig myself. I will seend a PR. For pool'ed ureq use, all that code makes sense.

[algesten]

Where/how did you call flush?

This is just me hacking around locally to see if I can get the certificates out as part of the Transport chain.

I think that I have to write code for ureq to do that. I'm also looking at the rustls config cache, and I don't want any of that: I want to provide the Rustls ClientConfig myself. I will seend a PR.

Yes, this requires bespoke code. In ureq3 we deliberately do not want rustls or any other 3rd party crate part of our public API. In ureq2 we had multiple situations of breaking changes in a 3rd party crate API that ureq exposed.

For ureq3, we made a common (simple) TLS API that unifies the most basic functions for the built-in rustls/native-tls.

For more advanced usage, the transport is pluggable, which means if there's a repeated need for many people to have deep rustls configurations, it's possible to publish a crate (ureq-rustls?) that has this feature. Similarly there could be ureq-mbedtls or any other transport crates.

[mcr]

Where/how did you call flush?

This is just me hacking around locally to see if I can get the certificates out as part of the Transport chain.

so, this could be a new fn on trait Connector. I would have called it "connect()", but that's already taken.
connect() should maybe have been called build() or new() or ...
so let's call it "go()" maybe?

For more advanced usage, the transport is pluggable, which means if there's a repeated need for many people to have deep rustls configurations, it's possible to publish a crate (ureq-rustls?) that has this feature. Similarly there could be ureq-mbedtls or any other transport crates.

My efforts to build my own Connector was stymied by trying to build ConnectionDetails from outside of ureq3.
I think that's a problem. I eventually ignored that problem (which made testing impossible), and I was able to create a "NoConnector", which is just like TcpConnector, but the socket is already open.
I tried hard to replicate RustlsConnector in a way that would let me put in the custom ClientConfig, but I failed.
(I wanted MbedtlsConnector, but there mbedtls reasons why this won't be occurring soon)
So I wound up with a
.unversioned_rustls_client_config(rustls_config)
which nobody wants.

[mcr]

so, this could be a new fn on trait Connector. I would have called it "connect()", but that's already taken. connect() should maybe have been called build() or new() or ... so let's call it "go()" maybe?

I guess it has to go on Transport.
I'm now trying to figure out how to get from Request to Transport.
I think I want to do:

        let mut res = agent.post(&uri.to_string());
        res.go();   // or whatever we call it
        // now poke at Verifier I created that captures peer certificate