Observing VPN clients

Most of the VPN protocols supported by OpenConnect are proprietary, undocumented, and in many cases overly-complex, perhaps intentionally obfuscated.

In order to add support for new protocols, and sometimes even to improve or update support for existing ones, it is often necessary to observe existing clients and servers in order to understand how they work.

Observing TLS/SSL connections

Modern VPN protocols almost always support a UDP-based transport for tunneled packets, e.g. DTLS for the Cisco AnyConnect protocol, or ESP for the GlobalProtect protocol. This is because TCP over TCP is very suboptimal in terms of performance. However, most VPN protocols also support TLS/SSL for connection initiation and as a fallback, due to its universal availability even in highly filtered or firewalled network environments. It is typically more straightforward and productive to start out by observing the TLS/SSL side of a new VPN protocol, and saving the UDP-based transport for later (See this discussion thread from the mailing list during the initial work on the GlobalProtect protocol.)

The content of a TLS/SSL connection is end-to-end encrypted and authenticated, and the session keys are normally kept only in memory by the applications using these connections. Therefore, special techniques are needed to decrypt traffic and observe the plaintext bytes-on-the-wire of any TLS-based communications protocol.

There are more-or-less 3 possible techniques for decrypting TLS-based communications and observing the plaintext. The first two shown here are quite simply, but not universally usable; the third is more complex but should be applicable to any TLS-based VPN protocol.

(1) Using SSLKEYLOGFILE

If your VPN client is based on a standard TLS library and does not disable this mechanism, it is likely the most straightforward way to decrypt TLS traffic.

Most TLS libraries (including OpenSSL, GnuTLS, and LibNSS) support an environment variable called SSLKEYLOGFILE, which will cause applications using those libraries to write "premaster secrets" for their TLS/DTLS sessions to a file. This makes it straightforward to automatically decrypt the traffic using the Wireshark network protocol analyzer. Basically, you should run the VPN client that you're observing with this environment variable set, run Wireshark on the same computer, and configure Wireshark to use the resulting log file; see Wireshark's TLS decryption documentation for details.

Note that some proprietary VPN clients disable the SSLKEYLOGFILE mechanism to prevent decryption of their traffic, but many developers overlook this especially in early releases of their client software.

(2) Using server's RSA private key

If you have access to the RSA private key of a VPN server supporting the protocol you're observing (likely because you administer such a VPN), and can use RSA key exchange, then you can provide this RSA private key to Wireshark and automatically decrypt TLS traffic to and from this server using Wireshark; again, see Wireshark's TLS decryption documentation for details and limitations.

This works because the RSA key exchange supported by TLS (re)uses the server's private key for encryption of the ephemeral session keys, unlike Diffie-Helman or Elliptic Curve Diffie-Helman exchanges which do not. (More detailed explanation on StackExchange.)

This lack of forward secrecy of session keys is indeed why RSA key exchange has been deprecated in TLS 1.2 and removed in TLS 1.3. However, most VPN servers and clients continue to support TLS 1.2 or older with RSA key exchange for backwards compatibility, so this is still a viable option for decrypting traffic as of 2022.

(3) MITM

If you cannot use either of the above mechanisms, then you will need to use the MITM technique to decrypt traffic to/from the VPN client you are observing.

The basic idea is to convince the two peers of the TLS connection (VPN client and server) that they are communicating directly with each other via an end-to-end encrypted TLS session, when in fact each of them is in fact communicating with an intermediary (the "man-in-the-middle"). The intermediary creates two separate TLS sessions: one between the real client and the intermediary, and one between the intermediary and the real server. The intermediary can inspect and modify traffic before relaying it between the the peers, who continue to believe that they are communicating directly with each other.

In order to MITM-capture the traffic from a typical proprietary VPN client:

  1. Use mitmproxy, running on Linux. (There are other similar tools available, but the rest of this document will assume you are running mitmproxy on Linux.)
  2. Set up a virtual machine and install the required operating and VPN client. (It is possible to do this using a physical system as well, but this will make transparent routing more complex.) We recommend:

  3. Run mitmproxy on the host system, and route traffic from the VM through it transparently, that is without setting an explicit proxy. (See below, for why an explicit proxy is unlikely to work.)

    With KVM/QEMU and user-mode networking, this consists of setting the guest system's IPv4 gateway to 10.0.2.2 and setting up iptables rules on the Linux host to redirect traffic. (Described in more detail in mitmproxy's guide to transparently proxying virtual machines).

  4. Browse to the URL https://mitm.it on the VM; this is a special domain intercepted by mitmproxy itself.

    The web page shown will help you verify that (a) the VM's traffic is really passing through MITMproxy and (b) the VM's operating system trusts mitmproxy's certificates.

  5. Run the VPN client on the VM, and capture and study its traffic using mitmproxy (or mitmdump) on the host.

  6. The client and/or server may employ various mechanisms to detect MITM and shut down the MITM'ed connection. Trick, bludgeon, or confuse the client and server into ignoring this; see below for specific examples.

    This usually works by sending a copy or hash/fingerprint of the server's TLS certificate as part of the in-band protocol data, and checking for differences from the certificate exchanged in the TLS handshake.

  7. Repeat these steps iteratively to understand how the VPN client's interaction with the VPN server works.

Defeating anti-MITM measures

Many proprietary VPN clients will ignore explicit system/browser proxy settings and attempt to establish direct connections to VPN servers, which is why explicit proxy setups often will not work, and transparent proxying is required.

In cases where your VPN client/server detect MITM'ed connections and stop communicating (6), it's typically necessary to write a script which will detect and rewrite in-band protocol data containing the server's TLS certificate (or a hash/fingerprint of it). This requires a recent version of mitmproxy containing pull request #1935 which was contributed by OpenConnect developers for this purpose.

An example of such a script may be useful: gp_ssl_log.py. This script works with mitmproxy or mitmdump to modify traffic to/from with a GlobalProtect server; specifically it mangles the XML tag sent by GlobalProtect portal servers, which GlobalProtect's official client software uses to verify that certificates of servers it connects to. It can be used with mitmdump as follows:

  ./mitmdump -vvv --tcp $YOUR_GP_SERVER_IP -s /path/to/gp_ssl_log.py --ssl-version-server all --insecure

Documenting VPN protocols

Sometimes it's most satisfying and productive to simply start coding up support for a new protocol or feature in OpenConnect itself. However, if you don't understand the structure of the OpenConnect codebase well, this may make it difficult to explain and discuss the code with more experienced developers.

Often it is very useful to take a step back and explain how the VPN protocol works in more human-readable terms before implementing it in OpenConnect's codebase. Preparing a document that intersperses prose explanations with captured traffic can aid in communication. See PAN_GlobalProtect_protocol_doc.md, (prepared during the implementation of the GlobalProtect protocol in OpenConnect, and subsequently updated) for a complete example.

Ask for help

Ask on the mailing list for help with new protocol implementations, or show us your code via Merge Requests on GitLab.