Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 3 and pygobject port of Zenmap #2088

Closed
wants to merge 33 commits into from

Conversation

kulikjak
Copy link

@kulikjak kulikjak commented Jul 27, 2020

Hi; as part of our migration effort away from Python 2, I managed to port whole Zenmap to Python 3 and pygobject library. Except for some minor visual glitches (like windows being wider than in current version) and most likely several hidden bugs I wasn't able to find yet, it is fully functional and will most likely go into the next update of Solaris.

Its only big issue is that it is not Python 2 and PyGTK backward compatible. I don't think it's that big of a deal as both are obsolete today and many Linux distributions don't ship them anymore, but you might think otherwise. Python 2 compatibility can be most likely brought back with the addition of #1972, PyGTK compatibility would require much more effort.

This, together with #1807 and some setup.py and configure changes (I have those ready as well), should be enough to completely port Nmap to Python 3.

It still needs some work (mostly cleaning up), but I wanted to make this PR now so that you know that this exists and can make comments :).

EDIT: cleaning up is done

@kulikjak
Copy link
Author

kulikjak commented Aug 3, 2020

I cleaned and fixed all remaining issues I found, and I believe its ready for review. There are still some things that should be done, but its already a pretty big patch, and those are not necessary for this transition.

Done:

  • Migrate everything to Python 3
  • Migrate all PyGTK code to PyGObject
  • Migrate some deprecated Gtk stuff to newer alternatives (initializers, horizontal and vertical containers (e.g., HBox and VBox -> Box), Arrows, ...)
  • Remove no longer relevant code necessary for PyGTK backward compatibility

ToDo later:

  • Remove all deprecated Gtk code (Gtk.Alignment, Gtk.IconFactory & Gtk.IconSet, Gtk.Table, Gtk.UIManager, stock items and probably some other things as well)
  • move optparse to argparse
  • probably more stuff I am forgetting right now

@vmkalbskopf
Copy link

If I try compile the current state of the master branch, will I be able to use zenmap on Ubuntu 20.04?

@chmelarp
Copy link

I did some intensive tests of master-python3 branch on Ubuntu 20.04 and .10 both as root and ordinary user. It behaves like it did before :)
I give it a go to merge it into nmap master

@Stunkymonkey
Copy link

maybe also update the list of dependencies in /zenmap/README

@baryluk
Copy link

baryluk commented Jan 19, 2021

Just curious, any progress on finishing the review and merging this?

@chmelarp
Copy link

chmelarp commented Mar 1, 2021

Hey,
after some time using the https://github.com/kulikjak/nmap I found one bug there. Also, the Readme might be updated, as it is confusing ... as Stunkymonkey noted. Also, an icon is missing there on Ubuntu :)
Thanks!

Version: 7.80SVN
Traceback (most recent call last):
File "/home/p3/Projects/Kali/zenmap3/zenmap/radialnet/gui/RadialNet.py", line 880, in draw
self.__draw(context)
File "/home/p3/Projects/Kali/zenmap3/zenmap/radialnet/gui/RadialNet.py", line 270, in check_graph_status
return function(*args)
File "/home/p3/Projects/Kali/zenmap3/zenmap/radialnet/gui/RadialNet.py", line 1023, in __draw
self.__draw_node(context, node)
File "/home/p3/Projects/Kali/zenmap3/zenmap/radialnet/gui/RadialNet.py", line 1171, in __draw_node
stride, data = get_pixels_for_cairo_image_surface(icon)
File "/home/p3/Projects/Kali/zenmap3/zenmap/radialnet/gui/Image.py", line 165, in get_pixels_for_cairo_image_surface
data[j:j + FORMAT_RGBA] = array.array('B', [r, g, b, a])
TypeError: an integer is required (got type str)

@kulikjak
Copy link
Author

Hi, good to know that somebody is using it :).

Can you please let me know how to reproduce this? I am unable to do so, and looking at the code, I don't see any obvious issue there. Thanks!

Considering the Readme, you are right, but I don't really want to do that before somebody maintaining nmap expresses interest in this PR and the whole Python 3 migration.

@surfzoid
Copy link

surfzoid commented May 4, 2021

Hi, i'm trying it for zenmap, mageia 8 have nmap for python 3 and no more py 2.
If i run zenmap directly from the git clone, it is okay except few warning :
https://paste.gnome.org/pnoaw5azd
trying to install on my system with sudo ./setup.py install
but :
https://paste.gnome.org/panvym1ln

@surfzoid
Copy link

surfzoid commented May 4, 2021

Another thing, the zenmap README is obsolete, with python 2

@kulikjak
Copy link
Author

kulikjak commented May 4, 2021

Yes, setup.py is not changed in this PR because there is another one doing that (I am not sure if they are compatible now, we will have to merge all these changes once this gets pulled). README is also not changed - that one can wait, I guess ;).

@surfzoid
Copy link

surfzoid commented May 4, 2021

Yes, setup.py is not changed in this PR because there is another one doing that (I am not sure if they are compatible now, we will have to merge all these changes once this gets pulled). README is also not changed - that one can wait, I guess ;).

since july 2020 did you get feed back from official nmap? do you know if it will be merged soon?

@kulikjak
Copy link
Author

kulikjak commented May 4, 2021

since july 2020 did you get feed back from official nmap? do you know if it will be merged soon?

Yes, it's discussed here (along with several other PRs for this transition): #1176. Just recently, they mentioned that it is a priority, but there are more pressing things to address first.

@Borim7
Copy link

Borim7 commented Oct 23, 2021

Thanks for the effort it is working fine.

I found 2 small points:

  • shebang line of zenmap file still points to python, which on most system is python2
  • When start zenmap the first time, the window size is a little bit to small to show all components. The list of scanned devices is only partly visible.
    • After resizing everything is displayed. As the window size is memorized, it only happen on the first startup.

@Borim7
Copy link

Borim7 commented Oct 23, 2021

Your port crash when clicking on filter host below the host list with following traceback:

Version: 7.80SVN
Traceback (most recent call last):
  File "/Daten/code/nmap/zenmap/zenmapGUI/ScanInterface.py", line 283, in filter_toggle_toggled
    self.filter_hosts(self.filter_bar.get_filter_string())
  File "/Daten/code/nmap/zenmap/zenmapGUI/ScanInterface.py", line 300, in filter_hosts
    start = time.clock()
AttributeError: module 'time' has no attribute 'clock'

@kulikjak
Copy link
Author

Thanks for the reports, I will look into those.

I migrated this using Python 3.7 and time.clock() was removed in 3.8, which I guess is why you are seeing the error I don't.

@ferdnyc
Copy link

ferdnyc commented Nov 26, 2021

@kulikjak

This is really promising! One thing I'd suggest is running your code with PYTHONWARNINGS=always set in the environment, to enable (in particular) DeprecationWarnings for future changes to Python or other libraries. (The breakage due to integer promotion being removed from Python 3.10 has shown how important that is.) Currently the branch has quite a lot of them:

$ PYTHONWARNINGS=always sudo python3 ./zenmap
[path]/nmap/zenmap/zenmapGUI/Icons.py:186: PyGTKDeprecationWarning: Gtk.IconSet(pixbuf) has been deprecated. Please use: Gtk.IconSet.new_from_pixbuf(pixbuf)
  iconset = Gtk.IconSet(pixbuf=pixbuf)
/usr/lib/python3.10/site-packages/gi/overrides/Gtk.py:1604: DeprecationWarning: Gtk.IconSet.new_from_pixbuf is deprecated
  iconset = Gtk.IconSet.new_from_pixbuf(pixbuf)
[path]/nmap/zenmap/zenmapGUI/Icons.py:187: DeprecationWarning: Gtk.IconFactory.add is deprecated
  iconfactory.add(key, iconset)
[path]/nmap/zenmap/zenmapGUI/Icons.py:189: DeprecationWarning: Gtk.IconFactory.add_default is deprecated
  iconfactory.add_default()
<frozen importlib._bootstrap>:671: ImportWarning: DynamicImporter.exec_module() not found; falling back to load_module()
[path]/nmap/zenmap/zenmapGUI/ScanToolbar.py:194: DeprecationWarning: Gtk.Container.set_focus_chain is deprecated
  self.target_entry.set_focus_chain((self.target_entry.get_child(),))
[path]/nmap/zenmap/zenmapGUI/ScanOpenPortsPage.py:204: DeprecationWarning: Gtk.ScrolledWindow.add_with_viewport is deprecated
  self.add_with_viewport(self.host)
[path]/nmap/zenmap/zenmapGUI/NmapOutputViewer.py:215: DeprecationWarning: Gdk.Color.to_string is deprecated
  "foreground", Gdk.Color(*text_color).to_string())
[path]/nmap/zenmap/radialnet/bestwidgets/labels.py:158: DeprecationWarning: Gtk.Misc.set_alignment is deprecated
  self.set_alignment(0, 0.50)
[path]/nmap/zenmap/radialnet/bestwidgets/expanders.py:146: DeprecationWarning: Gtk.Alignment.new is deprecated
  self.__alignment = Gtk.Alignment.new(0, 0, 1, 1)
[path]/nmap/zenmap/radialnet/bestwidgets/expanders.py:147: DeprecationWarning: Gtk.Alignment.set_padding is deprecated
  self.__alignment.set_padding(12, 0, 24, 0)
[path]/nmap/zenmap/radialnet/bestwidgets/boxes.py:219: DeprecationWarning: Gtk.Table.set_row_spacings is deprecated
  self.set_row_spacings(spacing)
[path]/nmap/zenmap/radialnet/bestwidgets/boxes.py:220: DeprecationWarning: Gtk.Table.set_col_spacings is deprecated
  self.set_col_spacings(spacing)
/usr/lib/python3.10/site-packages/gi/overrides/Gtk.py:1545: DeprecationWarning: Gtk.Table.attach is deprecated
  Gtk.Table.attach(self, child, left_attach, right_attach, top_attach, bottom_attach, xoptions, yoptions, xpadding, ypadding)
[path]/nmap/zenmap/radialnet/gui/ControlWidget.py:759: PyGTKDeprecationWarning: Using positional arguments with the GObject constructor has been deprecated. Please specify keyword(s) for "stock_id" or use a class specific constructor. See: https://wiki.gnome.org/PyGObject/InitializerDeprecations
  self.__force = Gtk.ToolButton(Gtk.STOCK_REFRESH)
[path]/nmap/zenmap/radialnet/gui/ControlWidget.py:892: PyGIDeprecationWarning: GObject.timeout_add is deprecated; use GLib.timeout_add instead
  GObject.timeout_add(REFRESH_RATE, self.__update_options)
[path]/nmap/zenmap/radialnet/bestwidgets/buttons.py:145: DeprecationWarning: Gtk.Image.set_from_stock is deprecated
  self.__image.set_from_stock(stock, self.__size)
[path]/nmap/zenmap/zenmapGUI/MainWindow.py:234: DeprecationWarning: Gtk.ActionGroup.new is deprecated
  self.main_action_group = Gtk.ActionGroup.new('MainActionGroup')
/usr/lib/python3.10/site-packages/gi/overrides/Gtk.py:279: DeprecationWarning: Gtk.ActionGroup.add_action_with_accel is deprecated
  self.add_action_with_accel(action, accelerator)
[path]/nmap/zenmap/zenmapGUI/MainWindow.py:432: DeprecationWarning: Gtk.ActionGroup.list_actions is deprecated
  for action in self.main_action_group.list_actions():
[path]/nmap/zenmap/zenmapGUI/MainWindow.py:433: DeprecationWarning: Gtk.Action.set_accel_group is deprecated
  action.set_accel_group(self.main_accel_group)
[path]/nmap/zenmap/zenmapGUI/MainWindow.py:434: DeprecationWarning: Gtk.Action.connect_accelerator is deprecated
  action.connect_accelerator()
/usr/lib/python3.10/site-packages/gi/overrides/Gtk.py:406: DeprecationWarning: Gtk.UIManager.insert_action_group is deprecated
  return Gtk.UIManager.insert_action_group(self, buffer, length)
/usr/lib/python3.10/site-packages/gi/overrides/Gtk.py:403: DeprecationWarning: Gtk.UIManager.add_ui_from_string is deprecated
  return Gtk.UIManager.add_ui_from_string(self, buffer, length)
[path]/nmap/zenmap/zenmapGUI/MainWindow.py:507: DeprecationWarning: Gtk.UIManager.get_widget is deprecated
  menubar = self.ui_manager.get_widget('/menubar')

...That's all just between launching zenmap and the main window appearing.

@ferdnyc
Copy link

ferdnyc commented Nov 26, 2021

Speaking of window size, when you click "Details" in the upper-right corner of the output area (which is enabled once you've run at least one scan), the window that pops up is comically narrow, and its contents don't adjust if you widen it. (It unfortunately doesn't remember the new size, either, so it goes back to super-narrow every time.)

...Same crazy-narrow formatting in the "Host Details" tab of the main window, in fact.

@kulikjak
Copy link
Author

Thanks @ferdnyc.

Going through the code with PYTHONWARNINGS=always is something I didn't consider; thanks for the suggestion! It's a pretty long time since I originally created this PR, so there might be some other things that need fixing as new Python runtimes are coming out. Also, there were several new additions to the Zenmap code which should be incorporated as well.

We also have GTK4 now, but I guess it's still to early to migrate directly to that.

I will look into that in the following days (including the weird default sizes).

@ferdnyc
Copy link

ferdnyc commented Jan 15, 2022

@kulikjak
I have some partial fixes for the layout stuff I can see about getting into shape to open a PR against your PR (one of my favorite git tricks), it might help some.

The larger issue is that the entire GUI is based on UIManager... which is already deprecated in its entirety, even for Gtk3. Ideally, everything would be (or, to borrow a tense from Douglas Adams, willan have been) ported to GtkApplication. But that means migrating all of the layouts, the Menus, the Actions... kind of "everything", gui-wise.

So I'm not sure a move to Gtk4 would be all that premature... it might actually make more sense than doing a migration within Gtk3, seeing as it's kind of a UI-design cul-de-sac at this stage.

That being said, I haven't really looked into how big a difference there is between them (I haven't really looked into Gtk4 at all), so I don't have a feel for the relative difficulty/effort each path would require.

@kulikjak
Copy link
Author

@ferdnyc

@kulikjak I have some partial fixes for the layout stuff I can see about getting into shape to open a PR against your PR (one of my favorite git tricks), it might help some.

Thanks, that would be great!

The larger issue is that the entire GUI is based on UIManager... which is already deprecated in its entirety, even for Gtk3. Ideally, everything would be (or, to borrow a tense from Douglas Adams, willan have been) ported to GtkApplication. But that means migrating all of the layouts, the Menus, the Actions... kind of "everything", gui-wise.

Yeah, I know about that... When I started the migration, my goal was to do only changes necessary to run with gtk3 and python3, which is why I left many deprecation warnings behind. I didn't want to rewrite everything in one PR. But since this still isn't merged and GTK is moving fast, maybe it's time to look into that as well.

So I'm not sure a move to Gtk4 would be all that premature... it might actually make more sense than doing a migration within Gtk3, seeing as it's kind of a UI-design cul-de-sac at this stage.

That being said, I haven't really looked into how big a difference there is between them (I haven't really looked into Gtk4 at all), so I don't have a feel for the relative difficulty/effort each path would require.

I am more worried from the OS/distro point of view - GTK 4 is still pretty new and not everybody might have it available, especially considering that Nmap is used by a very diverse list of OSes/distros. And GTK 3 will most likely still be here for years to come.


I guess the best thing is to ask maintainers what they prefer.

@fyodor @bonsaiviking - when the time comes for this PR, would you rather prefer a smaller change that just makes Zenmap compatible with GKT3 and Python 3 (that said, it's still already 2000 lines of changes) or a bigger change where all deprecation warnings and legacy code are taken care of? And should we go directly to GTK4, or is it still too early from your point of view?

@darkbasic
Copy link

darkbasic commented Jan 18, 2022

I think that most distros (like Ubuntu 22.04 LTS), even if they will stick to gtk3 apps when possible, they will still ship gtk4 alongside. By the time this gets released it will probably be better to have GTK4 IMHO.

@jerji
Copy link

jerji commented Jan 22, 2022

Hey, I was running a scan and when I double clicked on a host zenmap crahsed.
Here's the error:

Traceback (most recent call last):
  File "/home/angel/Downloads/nmap/zenmap/radialnet/gui/RadialNet.py", line 812, in draw
    self.__draw(context)
  File "/home/angel/Downloads/nmap/zenmap/radialnet/gui/RadialNet.py", line 202, in check_graph_status
    return function(*args)
  File "/home/angel/Downloads/nmap/zenmap/radialnet/gui/RadialNet.py", line 955, in __draw
    self.__draw_node(context, node)
  File "/home/angel/Downloads/nmap/zenmap/radialnet/gui/RadialNet.py", line 1103, in __draw_node
    stride, data = get_pixels_for_cairo_image_surface(icon)
  File "/home/angel/Downloads/nmap/zenmap/radialnet/gui/Image.py", line 97, in get_pixels_for_cairo_image_surface
    data[j:j + FORMAT_RGBA] = array.array('B', [r, g, b, a])
TypeError: an integer is required (got type str)

Seems like changing line 93 of Image.py from a = '\xff' to a = 255 fixes the bug

@kulikjak
Copy link
Author

kulikjak commented Aug 25, 2022

So, I rebased everything onto the latest master incorporating other changes pushed since then and fixed minor regressions. If anybody from upstream is interested, it's ready again! ;)

@jerji thanks for the report - it's fixed as well.

@surfzoid
Copy link

surfzoid commented Nov 3, 2022

Hi
i'm trying to build rpm, i cannot use zip download in spec file because is not versioned

@surfzoid
Copy link

surfzoid commented Nov 4, 2022

Hi,
RPM is build and work
https://mageialinux-online.org/repository/8/x86_64/media/core/
For further update versionned zip should be nice.

@kulikjak
Copy link
Author

kulikjak commented Nov 6, 2022

I am not sure that is related to this Pull Request? But maybe I am just missing something...

cclauss added a commit to cclauss/nmap that referenced this pull request Nov 7, 2022
Prove the Python syntax errors as discussed in
* nmap#91
* nmap#342
* nmap#624
* nmap#665
* nmap#666
* nmap#1176
* nmap#1484
* nmap#1807
* nmap#1972
* nmap#2088 
* nmap#2279
* nmap#2287
* nmap#2446
* nmap#2493
* nmap#2522
And many more...
cclauss added a commit to cclauss/nmap that referenced this pull request Nov 7, 2022
Prove the Python syntax errors as discussed in
* nmap#91
* nmap#342
* nmap#624
* nmap#665
* nmap#666
* nmap#1176
* nmap#1484
* nmap#1807
* nmap#1972
* nmap#2088 
* nmap#2279
* nmap#2287
* nmap#2446
* nmap#2493
* nmap#2522
And many more...
@nmap-bot nmap-bot closed this in 24b2631 Dec 7, 2022
@kulikjak
Copy link
Author

kulikjak commented Dec 8, 2022

Thank you!

@dmiller-nmap dmiller-nmap mentioned this pull request Apr 17, 2023
pghmcfc added a commit to pghmcfc/nmap that referenced this pull request Mar 1, 2024
Installation of translations hasn't worked since the conversion to
Python 3 (nmap#2088). That's because os.walk is not a drop-in replacement
for os.path.walk. This tweak to setup.py should address that (fixes nmap#2718).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

10 participants