A rendering engine (also known as layout engine) is a software behind the Web browser to display web pages. In other words, it is not a browser but a rendering engine who displays web pages written in mark up languages in a way how you see with the Web browsers. Without a rendering engine, web pages are just a simple text form with a lot of tags and words.
To help you understand better, the below table explains a relation with a kind of rendering engines to each major browsers.
|Lunascape||Gecko & Trident & Webkit|
For more details on rendering engine, please refer to Wikipedia.
You can also view a list of rendering engines available in Wikipedia.