How can I adjust the nits output in HDR mode?

You can use Contrast to adjust the peak brightness in SDR and HDR modes, and Brightness to adjust the grayscale from black-to-white.

Related Menu Pathways:

Options > Display > Brightness > HDR [Off/On]

Options > Display > Brightness > Contrast

Options > Display > Brightness