So after some more trial and error, I was able to display static content by including the following:

location ~* \.(?:ico|css|js|gif|jpe?g|png|woff2|map)$ {
# Some basic cache-control for static files to be sent to the browser
expires max;
add_header Pragma public;
add_header Cache-Control "public, must-revalidate, proxy-revalidate";
}

and adding a root:

root /var/www/html/;

However, this breaks the Authelia authentication server which lives on /

So, what I REALLY need to figure out is how to create a static location like above but instead of ~* have it specific for a location like /admin

How would I go about accomplishing that??