I'm a bit confused as to what could be causing this. Can you reduce the scan to a smaller set that still produces the error? For example, do a port scan for 5900, then do your --script vnc-brute against each of the discovered systems separately. Does one of them cause the error, or do they all finish?

If you can get it down to just one target that causes the error, then further try to reduce the problem:

• Does it happen with --script-args unpwdb.passlimit=5
• Does it happen with --script-args unpwdb.timelimit=5s

Finally, what is the output of nmap --version?

The nmap version I'm running:

Nmap version 7.60 ( https://nmap.org )
Platform: x86_64-pc-linux-gnu
Compiled with: liblua-5.3.3 openssl-1.1.0g libssh2-1.8.0 libz-1.2.8 libpcre-8.39 nmap-libpcap-1.7.3 nmap-libdnet-1.12 ipv6
Compiled without:
Available nsock engines: epoll poll select

I had to do the IP ranges from 30 at a time to 10 and then to 5 to stop getting the error. I couldn't bring it down to 1 IP that was the issue. I tried the above script-args and I got the same issue.