My way my road

Wireshark IKEV1 decrypt and ESP decrypt

>>>>>>>from https://ask.wireshark.org/questions/12019/how-can-i-decrypt-ikev1-andor-esp-packets <<<

IKEv1 Decryption
First of all: Wireshark 1.8.0 implements only 3DES and DES for IKEv1 decryption (same for version 1.6.8).

See: epan\dissectors\packet-isakmp.c: decrypt_payload()

If you want do decrypt any other algorithm, the dissector needs to be extended (Volunteers are welcome!). You can file an enhancement request for this at https://bugs.wireshark.org, possibly with a link to this question.
To get the required IKEv1 parameters for the dissector (Initiator's COOKIE and Encryption Key) you need debug output from your IPSEC implementation.

I tested with strongSwan 4.4 on Linux and with this capture file (with the capture file and the data provided in this answer, you can try it yourself). To get the value of "enc key" in the log, you need at least this debug option: --debug-crypt.
Look for ICOOKIE and enc key in the Pluto debug log.

gw205:/# ps auxww | grep pluto
root     24522  0.0  0.3  12572  3488 ?        Ss   15:46   0:00 /usr/libexec/ipsec/pluto --nofork --debug-raw --debug-crypt --debug-parsing --debug-emitting --debug-control --nocrsend --nat_traversal --keep_alive 60

strongSwan ipsec debug log:

2012:07:23-16:40:04 gw205 pluto[24522]: | 
2012:07:23-16:40:04 gw205 pluto[24522]: | *received whack message
2012:07:23-16:40:04 gw205 pluto[24522]: | creating state object #12 at 0x9fd77a8
2012:07:23-16:40:04 gw205 pluto[24522]: | ICOOKIE:  c6 d1 45 92  85 15 0c 7e
2012:07:23-16:40:04 gw205 pluto[24522]: | RCOOKIE:  00 00 00 00  00 00 00 00
2012:07:23-16:40:04 gw205 pluto[24522]: | peer:  c0 a8 8c c8
2012:07:23-16:40:04 gw205 pluto[24522]: | state hash entry 22

2012:07:23-16:40:14 gw205 pluto[24522]: | Skeyid_e:  b0 16 81 21  5f 16 20 23  03 18 6d 28  14 dc 56 86
2012:07:23-16:40:14 gw205 pluto[24522]: |   ca 5a 47 33
2012:07:23-16:40:14 gw205 pluto[24522]: | enc key:  44 9e 82 9e  a9 66 d4 21  fb cb 86 bd  7a d9 2e 86
2012:07:23-16:40:14 gw205 pluto[24522]: |   5a ba b1 5b  aa 5c 67 2a
2012:07:23-16:40:14 gw205 pluto[24522]: | IV:  dc f8 5e 03  f2 76 ab b9  89 e6 ae ff  46 a9 58 16
2012:07:23-16:40:14 gw205 pluto[24522]: |   f4 96 86 25

HINT: If you use any other IPSEC implementation please read the manual how to get that information.
Extract the values of ICOOKIE and 'enc key' WITHOUT spaces. HINT: The "enc key" spans two lines!!
ICOOKIE: c6d1459285150c7e
Enc Key: 449e829ea966d421fbcb86bd7ad92e865abab15baa5c672a
Use those values for

Edit -> Preferences -> Protocols -> ISAKMP -> IKEv1 Decryption Table:

Test File: ipsec.pcap
Result without decryption:
IKEv1 main mode - no decryption

Result with decryption:
IKEv1 main mode - WITH decryption

ESP Decryption
To decrypt ESP packets with Wireshark 1.8.0, you need again debug output from your IPSEC implementation. For Linux and strongSwan, you'll get that information with this command:

ip xfrm state

Output:

gw205:/ # ip xfrm state
src 192.168.140.200 dst 192.168.140.205
        proto esp spi 0x0879355b reqid 16421 mode tunnel
        replay-window 32 flag noecn nopmtudisc af-unspec
        auth hmac(sha1) 0xb8dd42a1c505bed19c2bf23cef00e5d8223c2a5b
        enc cbc(des3_ede) 0xae76ea430b10c72c882c4aeab2283444c54f913d87f5e109
src 192.168.140.205 dst 192.168.140.200
        proto esp spi 0x1c0d7b38 reqid 16421 mode tunnel
        replay-window 32 flag noecn nopmtudisc af-unspec
        auth hmac(sha1) 0xc364660133b04a4f20e52000dbe4a6ba154c09c1
        enc cbc(des3_ede) 0x39e87c9ca500616b36f2f0d3c7fb688621d7bbf31414abbd

Use those values for the ESP dissector parameters, as shown in the following screenshots. HINT: Take care not to add a space at the end of any parameter (SPI, key, etc.) as decryption will not work in that case.
First enable ESP decryption.