Is it reliable? It doesn’t get more basic than this. Nothing else matters if your site goes down, if there are glitches in the system or other problems. Look for a hosting company with a 99% or better up-time guarantee.
Does it run the software you need? Modern sites built using programs like WordPress need up-to-date software like PHP5 and mySQL databases.
What is the company’s customer service like? Do you get a real person when you call? Or just an automated system. How long are hold times? How fast does support respond to email?
Are there tools that make it easy for beginners? Features like 1-click software installation can be important if you’re new to the technology.
Is it fast? Aside from making sure your site visitors have a good experience, if your website is sluggish, the search engines tend to rank it negatively.
Are packages available that suit your needs, or is it a one-size-fits-all proposition. You might need a big hosting account, if so get one! But if not, why pay for more than you need?