Proxying HTTPS

A basic HTTP proxy is trivial to write, if you aren't concerned with preserving the literal message, you can simply glue existing server and client together.

My first attempt I tried this using Hunchentoot as a server and Dexador as a client.

This is not a very good strategy, as it gives no guarantee that the message you send is the same as the one you recieved. These things might not be preserved, for example:

• Header order
• Whitespace
• Case

You will also have to do a lot of work to smooth out the differential between the server and client.

Create an HTTP reader and writer that preserves raw messages.

I used http for reading and writing HTTP messages, as well as providing a basic multithreaded TCP server. This works better for my intended goal but it is not easy to completely implement a reader for messages.

Issues arise when encodings are used (see Accept-Encoding header) as a normal char stream cannot handle everything it might recieve. I have yet to solve this but I believe the solution is to use a binary stream and the functions provided by chunga to read from it. For now I am working around this issue by stripping the `Accept-Encoding` header from requests before forwarding them.

Other issues I forsee will be the completeness of my HTTP reader when handling different types of messages where the body size is not determined by the normal `Content-Length` header.

If my second strategy doesn't work out, I will probably decide to modify existing HTTP libraries to preserve raw messages. fast-http by Fukamachi seems like a good candidate as long as I am comfortable with part of the program being in C.

I need to be able to handle SSL messages, since in practical use, almost every message is encrypted. This should not add much complexity to the program but I need to figure out how to properly handle the HTTP `Connect` method.

The optimal UI, (in my opinion), is Emacs-style, where every aspect of the program is introspectable and modifiable to the user. Commands are defined and then bound to keys for the user to invoke.

In contrast, most popular apps right now are QT/GTK/React apps where actions are done by clicking and navigating menus. I believe this is very slow and inefficient if you already know what you want to accomplish, but it is a good interface for exploring new things and being able to see some of what is available to use.

The solution to this conflict would be to build the program Emacs-style, and then create an optional facade to give users who want it a "clicky" interface.

For now, I am going to write the UI as a Lem "plugin". Lem has all the functionality for Emacs-style interfaces built-in, and since it is an interactive Lisp program, I can run the proxy server directly in Lem.

This is essential because the proxy has to do the SSL handshake with the client and pretend to be the host that you are proxying to. mkcert is a handy tool that automatically creates a certificate authority, installs it in your browser, and creates certificates for hosts.

To successfully complete the SSL handshake, you need to have a certificate for each host that the client wants to talk to. With mkcert, this is simple and we can shell out to it to create a new certificate if we don't already have the right one.

(defun certificate (host &key create)
  (let ((cert (cert-file (uiop:strcat host ".pem")))
        (key (cert-file (uiop:strcat host "-key.pem"))))
    (when create
      (unless (and (uiop:file-exists-p cert) (uiop:file-exists-p key))
        (uiop:with-current-directory ((cert-file ""))
          (uiop:run-program (uiop:strcat "mkcert " host)))))
    (values cert key)))

Now we can call (certificate "www.google.com" :create t) and get the certificate and key files, creating new ones if we didn't already have them.

Since a proper intercept proxy needs to handle both protocols, we need a way to do it on the same port. To do this, we can use RFC 2817.

In our connection handler, we need to make a special case if the request method is CONNECT, and then upgrade the protocol to HTTPS. It is really quite easy, all we have to do is write a 200 response to the socket, and then pass it to another connection handler that will do the SSL handshake.

Here is a simplified example of what my server looks like:

(defun write-ssl-accept (stream)
  "Send a 200 response to accept the connection."
  (http:write-response stream (make-instance 'http:response)))

(defun ssl-connection-handler (conn host)
  "Handle connections upgraded to SSL."
  (with-ssl-server-stream (stream conn (first (str:split ":" host)))
    (let* ((req (http:read-request stream :host host))
           (resp (send-request-ssl req :raw t :host host)))
      (http:write-response stream resp))))

(defun connection-handler (conn)
  "Handle incoming connections from the client."
  (let* ((stream (us:socket-stream conn))
         (req (http:read-request stream)))
    (if (string-equal :connect (http:request-method req))
        (let ((host (puri:render-uri (http:request-uri req) nil)))
          (write-ssl-accept stream)
          (ssl-connection-handler conn host))
        (let ((resp (http:send-request req :raw t)))
          (http:write-response stream resp)))))