Images are the most common answer by a wide margin. A photograph straight from a camera or phone can be several megabytes and is frequently displayed at a fraction of its actual dimensions, which means the browser downloads far more than it needs and then discards most of it. Resizing images to roughly the size they display at, compressing them, and using a modern format solves this. On many small business sites this single change halves load time.
The second common cause is everything loading before anything appears. Fonts fetched from another service, tracking scripts, chat widgets, and social embeds each add a request, and the page frequently waits for all of them before rendering. The fix is to audit what is actually there. Most sites carry at least one script from a tool nobody uses any more, and removing it is free.
Hosting matters less than people assume until it matters a great deal. Cheap shared hosting is usually adequate for a small site with modest traffic, and the difference between hosts is dwarfed by the difference between an optimised and unoptimised page. If your measured server response time is slow while everything else is clean, that is the point to look at hosting, and not before.
Set a target rather than chasing a perfect score. Under two seconds on a phone connection is a reasonable goal and is where most of the commercial benefit sits. Beyond that, the effort required rises sharply while the return falls, and time spent shaving a further fraction of a second is time not spent on the copy, which affects conversion more.
Measure from a phone on a mobile connection rather than from your own machine, because your experience of your own site is the least representative one available. You are on fast broadband, your browser has already cached the images and fonts, and you know exactly where everything is. A visitor arriving for the first time on a phone downloads everything fresh, and the difference between those two experiences is frequently several seconds. Most testing tools let you specify a mobile connection, and that is the number worth improving rather than the desktop score.