That reframing changes what you check. The question is not whether the site resizes, since almost every modern template does. It is whether the site is usable on a phone held in one hand by somebody standing outside in daylight. Those are different tests, and the second one fails far more often. Text too small to read without zooming, buttons too close together to tap accurately, forms requiring precision, and a menu that takes three taps to reveal what somebody wanted immediately.
Test it properly rather than by resizing your browser window, which shows you the layout without the conditions. Open your site on your own phone, on mobile data rather than wireless, and try to complete the thing you want visitors to do. Find your phone number and call it. Fill in the contact form. If any step makes you pause or squint, it is losing you people who will not persist as long as you just did.
Speed matters more on a phone than anywhere else because the connection is worse and the patience is shorter. The usual culprits are images uploaded at far larger dimensions than they display at, video that autoplays, and fonts or scripts loading from several places before anything appears. Compressing images alone typically produces the largest single improvement, and it costs nothing.
One thing worth checking that people rarely do is what happens when somebody taps your phone number or address. A number that is plain text rather than a link forces somebody to memorise and retype it, and an address that does not open a map does the same. Both are one line of code and both convert a moment of intent into a small obstacle, which on a phone is where people give up.
There is one measurement worth taking before deciding how much of this matters to you. Open your analytics and compare conversion rate on phones against desktop, segmented by device. Nearly every small business finds mobile converts at a lower rate, and the size of that gap tells you what the problem is worth. A mobile rate at half the desktop rate, on a site where most visitors arrive by phone, means the majority of your traffic is landing on the worse version of your business. That single number turns an abstract concern into a figure you can weigh against everything else competing for your attention.