IPsec overview

IPSec is a framework of symmetric encryption of IPv4 protocol which is good for VPN creation etc. Parts of the IPSec are Internet Key Exchange (binding all the pieces together), Diffie-Hellmanalgorithm (for secure key exchange) and symmetric pre-shared key or certificate based authentication. This main three components make IPSec very reliable and very difficult to break into (even though some of the algorithms are weak/vulnerable).

IPSec is using IKE in two phases (imaginatively names Phase I and Phase II). IKE itself is incorporating some more interesting techniques like isakmp (for automatic SA negotiation and maintenance) which is now the synonym for IKE on most current routers (even though it is just one of 4). So lets have a look at

IKE Phase I

In Phase I you can meet with terms like HMAC authentication, IKE security association (SA) or Diffie Helman. As all of these are par of the IKE phase I you do not have to worry much about it but still you should know of these  things before proceeding to the part about modes. So just very short explanation of these three terms:

  • HMAC is a way of authentication while using hash functions like MD5 or SHA-1
  • Diffie-Hellman algorithm is a way of encrypting traffic in a case both parties have just their counterpart’s public key
  • isakmp – protocol within IKE for SA negotiation and re-keying
  • IKE SA – is a list of encryption parameters negotiated and maintained by ike/isakmp in a form of a “table” called SA

Now where the necessary is written we can finally proceed to the more important stuff and that is  the modes. This phase can be done in three types of modes :

  • base mode (which is now not-to-be-seen anywhere except for some linux distros / old routers)
  • aggressive mode (which uses just three UDP packets for the whole phase I and is not widely used)
  • main mode (uses 6 packets in phase I and is generally most used and most recommended)

As I wrote in the brackets the main mode is the most used and that is for a reason. It takes 6 packets to complete phase I and thus if it fails on some step not much of CPU power is used whereas the aggressive mode is using only 3 packets so immediately after first received packet complex calculations have to be made. The other thing is that aggressive mode has more limited options of settings so it is not very good for fine tuning etc. As the Main mode is much more common I will continue with it. I will divide the whole process into three steps and describe what is happening:

Step A
host A  ———— IKE SA proposal —————->  host B
host A < ———— IKE SA proposal —————-  host B

Step B
host A  ———— IKE key exchange —————>  host B
host A <———— IKE key exchange —————  host B

Step C
host A  ———— authentication ID —————>  host B
host A  <———— authentication ID —————  host B

In step A both host sends the SA proposals with all the necessary information – if they match exactly the process will enter step B, where the DH takes place and public keys are exchanged. If the exchange is successful the algorithm will create symmetric encryption key for the authentication of peers. Now in this encrypted environment the actual key exchange (similar to eap) will take place (step C).

This is the end of phase one. There are minor differences in what is send in the packets according to the mode/authentication method used. I you would like to see all possibilities look at this pdf (and you could be interested in the whole page) with detailed IKE Phases descriptions.

After the exchange ends there could be an info packet sent to the remote peer.

IKE Phase II

So now you have authenticated endpoints for you communication and a secure way for exchanging any sensitive information so that is exactly what will be done in phase II Again in the IKE Phase II you have a mode called quick mode. In fact I was unable to find any other modes for phase two and to my knowledge it is used everywhere. But important notice – there is also different modes present they are IPSec modes (tunnel/transport) not IKE modes so do not mix these thing up!

Except for the mode there is one more thing – PFS – which stands for perfect forward secrecy and it is a way of authentication (DH again).  This feature causes that the peer authentication done in Phase I is discarded and new Diffie-Hellman based authentication is executed in the already encrypted environment. This behavior is rather paranoid but PFS switch (e.g. on firewalls) can cause a lot of troubles so just be aware of this option.

Within Phase II is also done the IPSec negotiation so let’s have a look on the packet flow:

host A  ————  IPSec SA proposal —————->  host B
host A  < ———— IPSecSA proposal —————-  host B
host A  —————–  ID HASH ———————–>  host B

In the SA proposal the authenticated peers exchange the actual information about the type of transformation (encapsulation), integrity algorithm and encryption (in SAs) and this exchange is ended by third packet which contains some parts created by isakmp commonly just proving the peer is there.

That is for he IKE exchange and phases. The next chapter will be just brief overview about the IPSec modes and some side info.

IPSEec Settings

In IPsec you have to know the following things:

  • transformations
  • modes
  • integrity algorithm

The transformations just says how the inbound packet will be handled – you have three options

  • AH – Authenticated Header -Protocol that ensures the authentication of the packet (does not encrypt it !)
  • ESP – Encapsulating Security Payload – this protocol ensures encryption, authentication and integrity of the original packet
  • AH+ESP – combination of both of the above first the ESP is used and later the AH

The combinations of the above (modes and transformations) you can find in detail here with some nice pictures and packet fields descriptions. The two last things you have to set is the integrity check algorithm and encryption. Usually it is MD5 or SHA1 for the first and AES or 3DES for the later.

All these settings are in sent overt to the peer in the IPsec SA and have to match. Sometimes there is a possibility of fall backs etc. But that is not really recommended.


Leave a Reply

Your email address will not be published. Required fields are marked *