It seems that there was two things I was doing wrong.
First, I included no "root" (I just assumed it would inherit the root from "/").
After I included the root, though, I was still not seeing the images, still getting 404's instead of 403's.
My root was:
/path/to/root/path/to/directory;
After I changed it to just:
/path/to/root;
it worked fine.
Final configuration was:
location /path/to/directory/ {
root /path/to/root;
valid_referers www.mydomain.org mydomain.org;
if ($invalid_referer) {
return 403;
}
}
I did not include 'none' or 'blocked' because it was more or less imperative that it could ONLY be accessed from my site. (I do realize that people can still spoof my site into their referrer header, so that it's not a "perfect" solution, but I couldn't risk masked headers (blocked) and no headers (none) being able to see it.)