We further progress along the line of Ref. [D. Lacroix, Phys. Rev. A 94, 043614 (2016)] where a functional for Fermi systems with anomalously large $s$-wave scattering length $a_s$ was proposed that has no free parameters. The functional is designed to correctly reproduce the unitary limit in Fermi gases together with the leading-order contributions in the $s$- and $p$-wave channels at low density. The functional is shown to be predictive up to densities $\sim 0.01{\mathrm{fm}}^{-3}$ that is much higher densities compared to the Lee-Yang functional, valid for $\rho <{10}^{-6}{\mathrm{fm}}^{-3}$. The form of the functional retained in this work is further motivated. It is shown that the new functional corresponds to an expansion of the energy in $\left({}^{}{a}_s{k}_F\right)$ and $\left({}^{}{r}_e{k}_F\right)$ to all orders, where $r_e$ is the effective range and $k_F$ is the Fermi momentum. One conclusion from the present work is that, except in the extremely low-density regime, nuclear systems can be treated perturbatively in $-{\left({}^{}{a}_s{k}_F\right)}^{-1}$ with respect to the unitary limit. Starting from the functional, we introduce density-dependent scales and show that scales associated with the bare interaction are strongly renormalized by medium effects. As a consequence, some of the scales at play around saturation are dominated by the unitary gas properties and not directly by low-energy constants. For instance, we show that the scale in the $s$-wave channel around saturation is proportional to the so-called Bertsch parameter ${\xi }_0$ and becomes independent of $a_s$. We also point out that these scales are of the same order of magnitude than those empirically obtained in the Skyrme energy density functional. We finally propose a slight modification of the functional such that it becomes accurate up to the saturation density $\rho \simeq 0.16{\mathrm{fm}}^{-3}$.