Adds libraries for Punycode and IDNA conversions. #958

rewanthtammana · 2017-08-03T15:37:15Z

No description provided.

dmiller-nmap

Ok, this is almost done! Now we're done bickering over rulesets and we're defining the interface for other NSE programmers and streamlining the code. Take a look at these changes and let me know if you have time to keep working on it.

dmiller-nmap · 2017-09-21T15:38:19Z

nselib/idna.lua

+-- Since this library is dependent on punycode and vice-versa, we need to
+-- import each of the libraries into another. This prevents idna from entering
+-- into a recursive loop.
+package.loaded["idna"] = _ENV


We can eliminate this confusion if we divide the functions cleanly between the two libraries. In punycode.lua we should only handle Punycode encoding and decoding of labels. These functions should be in idna.lua instead of punycode.lua:

Splitting names into labels based on RFC 3490 separators.

Unicode de/encoding. In fact, we could even make this the responsibility of the caller, and only accept tables of code points for Unicode inputs (i.e. never a string of encoded bytes)

Joining labels with "."

This has the potential to really simplify a lot of the code here. Most likely, scripts and other libraries will only require idna.lua, since that's the higher-level interface.

dmiller-nmap · 2017-09-21T15:42:59Z

nselib/idna.lua

+  decoded_tbl = concat_table_in_tables(decoded_tbl)
+
+  -- Regular expressions (RFC 3490 separators)
+  for index, cp in ipairs(decoded_tbl) do


This mapping is actually done in idnaMappings, so we don't need to do the additional checks here.

dmiller-nmap · 2017-09-21T15:47:45Z

nselib/idna.lua

+    for index, cp in ipairs(decoded_tbl) do
+      local lookup = idnaMappings[cp]
+      if type(lookup) == "table" then
+        if lookup.status == "deviation" then


There is no reason to keep looping over and over the code points. One loop is sufficient. Inside the loop:

do the lookup in idnaMappings

Replace the code point if it's a number

if it's a table, check the status. You can do if transitionalProcessing and lookup.status == "deviation" then to handle that case.

handle the code point as needed.

In other words, handle each code point exactly once. The output of idnaMappings is always final; if it's mapped, then it is mapped to one or more valid code points which do not need to be checked again.

dmiller-nmap · 2017-09-21T15:59:30Z

nselib/idna.lua

+-- Since this library is dependent on punycode and vice-versa, we need to
+-- import each of the libraries into another. This prevents idna from entering
+-- into a recursive loop.
+package.loaded["idna"] = _ENV


We can avoid this confusion by limiting punycode.lua to simply encoding and decoding labels instead of whole names. In other words, only idna.lua is responsible for:

Separating and joining names and labels on RFC 3490 separators.

Unicode-to-bytes encoding/decoding. In fact, we could require the caller to handle this instead, though that might lead to code duplication.

dmiller-nmap · 2017-09-21T16:26:49Z

nselib/idna.lua

+  decoded_tbl = concat_table_in_tables(decoded_tbl)
+
+  -- Regular expressions (RFC 3490 separators)
+  for index, cp in ipairs(decoded_tbl) do


These are already handled by idnaMappings, so no need to check separately for them.

Removed this piece of code.

dmiller-nmap · 2017-09-21T16:29:59Z

nselib/idna.lua

+-- @param checkBidi Boolean flag to represent if the input is of Bidi type.
+-- @param checkJoiners Boolean flag to check for ContextJ rules in input.
+-- @param useSTD3ASCIIRules Boolean value to represent ASCII rules.
+-- @param transitionalProcessing Boolean value.


Add default values to the NSEdoc and note any flags that are not supported. We can't rely on consumers to read the code; at best they will look at the documentation, so it must be correct.

dmiller-nmap · 2017-09-21T16:32:54Z

nselib/idna.lua

+  local inputString = unicode.encode(codepoints, encoder)
+
+  -- Checks for invalid domain codepoints and proceeds further.
+  if not match(inputString, "[^a-z0-9_@=+/%-%(%)%%]") then


What's the reference for these valid code points/characters? I'm surprised to see _@=+()% in there.

My bad, I deleted this piece of code.

dmiller-nmap · 2017-09-21T16:36:28Z

nselib/punycode.lua

+    -- incrementing `n` each time, so we'll fix that now:
+    if (floor(i / out) > maxInt - n) then
+      --error('overflow');
+      return nil, "Overflow exception occurred."


Are we checking for this and other punycode errors when we call these functions from idna.lua? We should be doing so and passing the error up to the caller.

punycode.mapLabels is the only function being used in idna.lua and this function is returned to the caller, so if there is an error the error will be returned to the caller along with nil as response value.

So, effectively it becomes the job of the caller to handle the errors being thrown.

dmiller-nmap · 2017-09-21T16:38:01Z

nselib/punycode.lua

+function encode_label(s, decoder)
+
+  local flag = false
+  local decoded_tbl = unicode.decode(s, decoder)


Remember, for simplicity we can require callers of punycode functions to already do the decoding and pass in a table of code points. After all, this is what idna.lua has after doing the mapping/validation, so it doesn't make sense to keep encoding and decoding between calls.

I thought decoding here will help the NSE developers who import this function exclusively.

dmiller-nmap · 2017-09-21T16:40:10Z

nselib/punycode.lua

+end
+
+-- Table of punycode test cases.
+local testCases = {


When you convert the libraries to move the domain splitting/joining and Unicode stuff to idna.lua, you'll have to change these test cases to each be a single label.

rewanthtammana · 2017-09-25T03:39:20Z

Almost all the requested changes have been done, please review the modified code.

dmiller-nmap

Just add missing requires for math and table to both, address these 2 typos, and commit! Thanks so much!

dmiller-nmap · 2017-09-27T03:34:17Z

nselib/idna.lua

+      -- To use this part of code, add disallowed_STD3_mapped and disallowed_STD3_valid
+      -- codepoints to idnaMappings.lua. For now, we ignore these because idnaMappings.lua
+      -- is set to support only for the latest version of IDNA.
+      if UseSTD3ASCIIRules then


This is miscapitalized and therefore not referring to useSTD3ASCIIRules from the parameters list.

dmiller-nmap · 2017-09-27T03:36:29Z

nselib/punycode.lua

+  local delimiterCodePoint = 0x002E
+  local delimiter = unicode.encode({0x002E}, encoder)
+
+  codepoints = unicode.decode(input, decoder)


Need to make this local

rewanthtammana added 8 commits August 3, 2017 21:01

Adds idna mapping table file.

bba5361

Adds punycode library.

d5ddb60

Adds idna library.

40893f8

Fixes test-cases indentation in IDNA library.

fad7ef7

Moves idnaMappings.lua to nselib/data/.

6f737e5

Requires file from new location.

a605da4

Moves XNV8 to disallowed category.

2a12664

Adds code for checking invalid domain codepoints.

c6b6c4a

rewanthtammana force-pushed the idna branch from bdba1fa to f95d04b Compare August 5, 2017 05:26

Updates regex.

6e42c4e

rewanthtammana force-pushed the idna branch from f95d04b to 6e42c4e Compare August 5, 2017 06:01

rewanthtammana added 6 commits August 5, 2017 12:14

Moves disallowed_STD3_* to disallowed category.

f398e4c

Adds comments regarding the extra code in idna.lua

afaf285

Converts single codepoint mapping to number instead of table.

9d0cd98

Modifies idna.lua to handle changes in idnaMappings.lua

e6cc718

Adds both libraries to unittest.lua.

f771f23

Updates documentation for both the libraries.

84e4c24

dmiller-nmap requested changes Sep 21, 2017

View reviewed changes

rewanthtammana added 6 commits September 24, 2017 18:36

Removes recursive importing of functions.

67911c2

Removes multiple looping with the use of if loops.

a480a55

Removes unwanted piece of code.

bc52be8

Updates punycode test cases

36ece77

Removes extra piece of code.

c13d419

Default values added to NSEdoc.

677d070

dmiller-nmap reviewed Sep 27, 2017

View reviewed changes

rewanthtammana added 3 commits September 28, 2017 00:31

Included local libraries.

dbbf97e

Fixes typo.

b072c2d

Variables localized.

fb881f4

nmap-bot closed this in fdc9b19 Sep 27, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds libraries for Punycode and IDNA conversions. #958

Adds libraries for Punycode and IDNA conversions. #958

rewanthtammana commented Aug 3, 2017

dmiller-nmap left a comment

dmiller-nmap Sep 21, 2017

rewanthtammana Sep 24, 2017

dmiller-nmap Sep 21, 2017

rewanthtammana Sep 24, 2017

dmiller-nmap Sep 21, 2017

rewanthtammana Sep 24, 2017

dmiller-nmap Sep 21, 2017

rewanthtammana Sep 24, 2017

dmiller-nmap Sep 21, 2017

rewanthtammana Sep 24, 2017

dmiller-nmap Sep 21, 2017

rewanthtammana Sep 24, 2017

dmiller-nmap Sep 21, 2017

rewanthtammana Sep 24, 2017

dmiller-nmap Sep 21, 2017

rewanthtammana Sep 24, 2017

rewanthtammana Sep 24, 2017

dmiller-nmap Sep 21, 2017

rewanthtammana Sep 24, 2017

dmiller-nmap Sep 21, 2017

rewanthtammana Sep 24, 2017

rewanthtammana commented Sep 25, 2017

dmiller-nmap left a comment

dmiller-nmap Sep 27, 2017

dmiller-nmap Sep 27, 2017

Adds libraries for Punycode and IDNA conversions. #958

Adds libraries for Punycode and IDNA conversions. #958

Conversation

rewanthtammana commented Aug 3, 2017

dmiller-nmap left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rewanthtammana commented Sep 25, 2017

dmiller-nmap left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment