I want know more about modbus over serial line and modbus over tcp/ip? when to use them?Is there any book for learning all communication protocols?

Main criteria for selection of communication protocol is this:
Speed of communication required, no of slave devices or nodes, compatability or device support, network topology redundancy etc.
check the specification of both protocol you'll get the answer. Modbus tcp/ip is modbus over Ethernet and is much faster.

Modbus protocol is very popular in Industry control system.

PLC & Remote IO or HMI could communicate with each other by modbus tcp/RTU very easily.

And modbus protocol is open to everyonet.

Because this protocol is opened, you can search it in google, and get some user guild line or manu to learn.

There are a large no. of Industrial Communication protocol. And they are defined for DCS, Instruments, different controller manufacturer like SIEMENS,GE FANUC, SCHNEIDER etc. And some times the choice depends on distance between the two nodes.

As mentioned above. A google search on "modbus manual" yields any number of pdf-results. What you should be aware of though is that most manufacturers have their own modbus versions and while MOST obey the "standard", there are plenty where you will need to basically reconfigure the message and decode it into something your PLC/HMI/SCADA will understand.

can any body let me know the difference between Modbus MS/TP and Modbus TCP/IP - apart from that MS/TP is over RS485 communication and TCP/IP is over lan network?the difference in the protocol structure used the security level difference in the protocol ?also pls let me know the advantage of BACnet over Modbus??

for saikat, if you want to find more detailed information, you can go to : www.modbus.org