So I was finally able to resolve this, but not in any way I've seen described anywhere else.

I set "net.ipv6.bindv6only = 1" in /etc/sysctl.conf and reloaded via "sudo sysctl -p". Then with the configuration "listen 80; listen [::]:80;" I now get my workers split evenly between ipv4 and ipv6 and now I can connect from either ip source.

nginx 2096 www-data 6u IPv4 286321 0t0 TCP *:http (LISTEN)
nginx 2096 www-data 7u IPv6 286322 0t0 TCP *:http (LISTEN)
nginx 2097 www-data 6u IPv4 286321 0t0 TCP *:http (LISTEN)
nginx 2097 www-data 7u IPv6 286322 0t0 TCP *:http (LISTEN)
nginx 2098 www-data 6u IPv4 286321 0t0 TCP *:http (LISTEN)
nginx 2098 www-data 7u IPv6 286322 0t0 TCP *:http (LISTEN)
nginx 2099 www-data 6u IPv4 286321 0t0 TCP *:http (LISTEN)
nginx 2099 www-data 7u IPv6 286322 0t0 TCP *:http (LISTEN)
nginx 2100 www-data 6u IPv4 286321 0t0 TCP *:http (LISTEN)
nginx 2100 www-data 7u IPv6 286322 0t0 TCP *:http (LISTEN)
nginx 2101 www-data 6u IPv4 286321 0t0 TCP *:http (LISTEN)
nginx 2101 www-data 7u IPv6 286322 0t0 TCP *:http (LISTEN)
nginx 2102 www-data 6u IPv4 286321 0t0 TCP *:http (LISTEN)
nginx 2102 www-data 7u IPv6 286322 0t0 TCP *:http (LISTEN)
nginx 2103 www-data 6u IPv4 286321 0t0 TCP *:http (LISTEN)
nginx 2103 www-data 7u IPv6 286322 0t0 TCP *:http (LISTEN)