Chọn phương án A.

Theo đề bài ta có
\begin{eqnarray*}
&f(x)&=x\left(1+\dfrac{1}{\sqrt{x}}-f'(x)\right)\\
\Leftrightarrow&f(x)&=x+\sqrt{x}-xf'(x)\\
\Leftrightarrow&f(x)+xf'(x)&=x+\sqrt{x}\\
\Leftrightarrow&\left[xf(x)\right]'&=x+\sqrt{x}
\end{eqnarray*}
Lấy nguyên hàm hai vế ta được $$\begin{aligned}
\displaystyle\int\left[xf(x)\right]'\mathrm{\,d}x&=\displaystyle\int\left(x+\sqrt{x}\right)\mathrm{\,d}x\\
\Leftrightarrow xf(x)&=\dfrac{x^2}{2}+\dfrac{2x\sqrt{x}}{3}+C.
\end{aligned}$$
Vì $f(4)=\dfrac{4}{3}$ nên ta có $$4f(4)=\dfrac{4^2}{2}+\dfrac{2\cdot4\sqrt{4}}{3}+C\Leftrightarrow C=-8.$$
Vậy $xf(x)=\dfrac{x^2}{2}+\dfrac{2x\sqrt{x}}{3}-8$.

Đặt $\begin{cases}
u=x^2-1\\ v'=f'(x)
\end{cases}\Rightarrow\begin{cases}
u'=2x\\ v=f(x)
\end{cases}$

Do đó $$\begin{aligned}
\displaystyle\int\limits_{1}^{4}\left(x^2-1\right)f'(x)\mathrm{\,d}x&=\left(x^2-1\right)f(x)\bigg|_1^4-\displaystyle\int\limits_{1}^{4}2xf(x)\mathrm{\,d}x\\
&=15f(4)-2\displaystyle\int\limits_{1}^{4}\left(\dfrac{x^2}{2}+\dfrac{2x\sqrt{x}}{3}-8\right)\mathrm{\,d}x\\
&=15\cdot\dfrac{4}{3}-2\left(\dfrac{x^3}{6}+\dfrac{4}{15}x^2\sqrt{x}-8x\right)\bigg|_1^4\\
&=20-\left[-\dfrac{128}{5}+\dfrac{227}{15}\right]=\dfrac{457}{15}.
\end{aligned}$$