Can I apply for a passport in Ireland / Malaysia while on holiday?

No, you must apply in the country you are normally resident in.

You should only apply to the Irish Embassy in Malaysia if you reside in Malaysia, or in the case of a lost/stolen passport while visiting here.