TCP handshake is a process of how two TCP/IP endpoints establishing a TCP connection. This process dominates how network applications begin to communicate to each other and how they finish the conversation. Understanding the TCP handshake is crucial when you want to develop network applications or troubleshoot network problems.

Below is an analysis on TCP connection handsaking which I did many years ago. Though this analysis is based on W. Richard Stevens's TCP/IP Illustrated series and Linux's BSD socket library, I think Windows are using the similiar concept in its winsock( via winsock.dll) as well.

States explanations (based on /usr/src/linux-2.2.12/net/ipv4/tcp.c):

SYN_SENT sent a connection request, waiting for ack

SYN_RECV received a connection request, sent ack, waiting for final ack in three-way handshake.

ESTABLISHED 3-way handshake complete, connection established.

FIN_WAIT1 our side has shutdown by closing close(), waiting to complete transmission of remaining buffered data.

FIN_WAIT2 all buffered data sent, waiting for remote to shutdown.

TIME_WAIT timeout to catch resent junk before entering closed, can only be entered from FIN_WAIT2 or CLOSING. Required because the other end may not have gotten our last ACK causing it to retransmit the data packet (which we ignore). Here has more detailed explanation of this state.

CLOSE_WAIT remote side has shutdown and is waiting for us to finish writing our data and to shutdown (we have to close() to move on to LAST_ACK).

LAST_ACK our side has shutdown after remote has shutdown. There may still be data in our buffer that we have to finish sending.

CLOSE socket is finished.

3-way handshake
Step 1
The client side initiates a TCP connection first by sending a packet containing with SYN flags set(by calling connect() function), this signals the listening socket (at the server side) that the client want to initiate a connection to it. The SYN packet also contains a sequence number, Initiate Sequence Number ISN. The purpose of this number is to keep tracking of the "conversion" between the client and the server. This is the core of the connection-oriented protocols. It is suggested that the ISN be as random as possible to prevent being spoofed. Many vendor have its different implementations because the true sequence number uses MD5 hashing algorithm, and this algorithm is not good at performance (This is also the reason why we only use MD5 to hash the checksum of the message in public-key algorithm, not the whole message). Linux, OpenBSD, and FreeBSD actually implement versions of RFC1948. Sun Microsystems and SGI includes RFC1948, seems but disabled by default.

Apart from the above, the SYN packet also contains the following parameters, they are crucial in the performance of the data transfer:

1. MSS (Max. Segment Size) is the max. amount of data that it is willing to accept in each TCP segment on this session. This is usually the max size of data that will be moved from the TCP layer to IP layer. To gain the best performance the should be set to the value such that the IP datagram fragmentation will not occur during the travel from the one end of the socket to the other end. The formula to calculate the best MSS: PMTU(Path MTU)- TCP header size(20 bytes) - IP header size(20 bytes also). For example, in ethernet network the MSS should be 1460 bytes (1500 -20- 20). The MSS of each direction can be different.

2. Window is the max # of data that a socket can send without ack. In general, high-speed or long-delay connection requires a larger window to obtain the max throughput.

Step 2
When the server received the SYN packet, it will response by sending packet with ACK = the client's sequence plus 1 (i.e ISN+1)if there is a listening socket for that port number. The server ACK packet also set the SYN. If the server side doesnt have the program listened to that port number, a packet with RST flag will be sent to client to indicate that the connection failed. Usually the connect() function returns ECONNREFUSED error code.

Step 3
If the server side responded with a packet with correct ACK (to prevent spoofing) and having SYN flag set, the client then responded by sent a packet with the server's sequence number plus 1. By this time, the TCP connection initiation complete. At this point client and server sides can send data to each one as they want until the close() at either sides have been called.

Points to note:
The 3-way handshake usually invisible to users and even socket programmer and ,as you can see, the handshake tasks at the client is totally done by connect() function and accept() at the server side.

4-way Termination handshake
In fact, TCP termination process is a less-to-be-known process where no official terminology for it. While some network programming guys call it 4-way termination handshake, it doesn't necessarily mean that 4 packets must be interchanged between the client and server. There are several ways to close an opening socket.

Step 1
The active close side, usually the client, send a packet with FIN flag set by calling close() function. The socket will be in FIN_WAIT_1. Since TCP is full-duplex connection, we say this a "Half-close". This indicates that we, the client, will not send data anymore, however, we still receive data from the other side.

Step 2
When the server side received the FIN packet, it, as usual, send an ACK back to the client which is the previous ACK number plus 1 as the FIN consume 1 byte of data. The socket the go from ESTABLISHED to the CLOSE_WAIT state while the client side go to the FIN_WAIT_2 state during the receipt of the server's ACK packet.

Step 3
The receipt of FIN invokes at the server side an end-of-file to the server application which causes the server send a FIN packet(by calling close() again) to the client and changes the state from ESTABLISHED to LAST_ACK.

Step 4
The client received the FIN packet from the server and send back an ACK packet to the server. The client changed the state to TIME_WAIT. Upon the receipt of ACK at the server, it change the state to CLOSED. This indicates that the socket close completely.