I am having problems with this patch. Firstly, the version in Gitlab is not the same as the installed version. It has a lot of updates so I am struggling a bit.

Secondly, in the file you've patched, there is a second similar code block at line 315:

                    try:

                        hc = httplib.HTTPSConnection('mirrorlist.clearos.com')

                        hc.request("GET", request, None, repo.http_headers)

                        hr = hc.getresponse()

                        if hr.status != 200:

                            raise Exception('unable to retrieve repository data.')



                        incpkgs = hr.read()

                        hc.close()

                        repo.setAttribute('includepkgs', incpkgs.split())

                    except:

                        pass

Does this also need modifying to something like:

                    try:

                        try:

                            hr = urllib2.urlopen('mirrorlist.clearos.com')

                        except urllib2.HTTPError, e:

                            raise Exception('unable to retrieve repository data.')



                        incpkgs = hr.read()

                        repo.setAttribute('includepkgs', incpkgs.split())

                    except:

                        pass

What does this bit of code do and do nested try statements work. I have tried this but the compiler is failing with an indentation issue and I suspect a different method is required to add the https headers. I think the compiler error I am getting is for nested try statements.

Your patch, on its own, seems to work but I am not sure if the second section of code needs modifying for the private repos.

Thanks for that. I'll try patching on Monday and I'll give you a link to the build.

Is this fix working with an authenticated proxy?

Nick Howitt wrote:

Hello Stephane,
If you can rpovide a patch for this, it would be great. You can clearly see the code as you've found it. The GitLab repo is here.

Reading the urllib docs, it looks like it does not support an authenticated proxy, although urllib2 may (it looks like there is a proxy handler example in the docs). If there is a proxy, and environment variable is set:

http_proxy=http://userassword@proxy_urlroxy_port

https_proxy=http://userassword@proxy_urlroxy_port

ftp_proxy=http://userassword@proxy_urlroxy_port

so this could be read. Otherwise setting are in /etc/clearos/upstream_proxy.conf (or /etc/profile.d/proxy.sh). My knowledge of python is a bit too rudimentary.

[edit]
It looks like the forum formatting is broken for my output. You need to do a:

env | grep proxy

to see what is set.
[/edit]

Hi Nick, urllib is broken when it comes to send HTTPS traffic to a proxy based on my tests. So indeed urllib2 needs to be used.
I didn't have time to check the Gitlab yet but here's the code change I did to get it working - access to MarketPlace now works fine through proxies :

--- /tmp/clearcenter-marketplace.py     2019-07-04 10:15:40.478490597 +0300

+++ /usr/lib/yum-plugins/clearcenter-marketplace.py     2019-07-12 17:35:16.250482904 +0300

@@ -6,6 +6,7 @@

 import re

 import httplib

 import urllib

+import urllib2

 import json

 import shutil

 import random

@@ -219,14 +220,12 @@



         request = "%s?%s" %(self.request, urllib.urlencode(params))



-        hc = httplib.HTTPSConnection(self.url)

-        hc.request("GET", request)

-        hr = hc.getresponse()

-        if hr.status != 200:

+        try:

+               hr = urllib2.urlopen("https://"+self.url+request)

+       except urllib2.HTTPError, e:

             raise Exception('unable to retrieve repository data.')



         buffer = hr.read()

-        hc.close()

         response = self.byteify(json.loads(buffer))



         if not response.has_key('code'):

I've been able to install the Radius application I wanted to check. It seems that it just allows to define Radius clients ..
The remaining actions (Radius users, allowed Mac-addresses, attributes) need to be set by editing FreeRadius configurations by hand ?
I thought it was more elaborate - as powerfull as Daloradius or PacketFence

Hello Stephane,
If you can rpovide a patch for this, it would be great. You can clearly see the code as you've found it. The GitLab repo is here.

Reading the urllib docs, it looks like it does not support an authenticated proxy, although urllib2 may (it looks like there is a proxy handler example in the docs). If there is a proxy, and environment variable is set:

http_proxy=assword@proxy_urlroxy_port" target="_blank">http://userassword@proxy_urlroxy_port

https_proxy=assword@proxy_urlroxy_port" target="_blank">http://userassword@proxy_urlroxy_port

ftp_proxy=assword@proxy_urlroxy_port" target="_blank">http://userassword@proxy_urlroxy_port

so this could be read. Otherwise setting are in /etc/clearos/upstream_proxy.conf (or /etc/profile.d/proxy.sh). My knowledge of python is a bit too rudimentary.

[edit]
It looks like the forum formatting is broken for my output. You need to do a:

env | grep proxy

to see what is set.
[/edit]

Dave Loper wrote:

The implementation of a the proxy support was a feature request from a Fortune 100 client that needed proxy support for their own network. We worked to implement within their spec and it was working when the project completed. Perhaps they had some exceptions that they put in that made other aspects work that I was not . But I can see where you saying and will investigate. Can you describe a little about the proxy you use?

Hi Dave. Proxy being used is Squid - but the issue would be the same whatever the proxy brand / solution used.
Right now ClearOS cannot be deployed in an environment where access to Internet is done only through proxies. Maybe some code was changed at a time and it used to work ...

The implementation of a the proxy support was a feature request from a Fortune 100 client that needed proxy support for their own network. We worked to implement within their spec and it was working when the project completed. Perhaps they had some exceptions that they put in that made other aspects work that I was not . But I can see where you saying and will investigate. Can you describe a little about the proxy you use?

Marketplace installs does not work through yum nor the web interface.
This is due to code in /usr/lib/yum-plugins/clearcenter-marketplace.py :

hc=httplib.HTTPSConnection(self.url)

httplib does not use proxy settings / variables.
Code should be updated to use maybe requests library - see https://stackoverflow.com/questions/53252991/cant-reach-ip-using-python-httplib
Maybe using urllib instead should be ok - by default, urlopen uses the environment variable http_proxy.

I guess Clearos was never tested in an isolated environment / where access to Internet is done through proxies.
It's a pity as some companies force Internet access through proxies ..
Can this be fixed for next release / through an update ?

So you are able to perform Marketplace installs using yum but not the web interface? Is that correct?

Hi Dave. Normally DNS resolution should be done by the proxy but as the request is not sent to the proxy, DNS resolution is done locally - and the ClearOS VM is in a test environement without DNS internet resolution - only the proxy has DNS access to Internet, hence the DNS issue reported back.
This is trange http_proxy and https_proxy variables are set correctly and even if I force a proxy setting in /etc/yum.conf access Clear Center MarketPlace is not done through the proxy.

I checked the /usr/lib/yum-plugins/clearcenter-marketplace.py code and it relies on urlgraber/httplib/yum modules.
yumRepo.py code apparently checks for proxy. For some strange reason Clearcenter Marketplace access is not done through the proxy but the other repos are ok :

yum repolist

Loaded plugins: clearcenter-marketplace, fastestmirror

ClearCenter Marketplace: fetching repositories...

ClearCenter Marketplace: [Errno -2] Name or service not known

Loading mirror speeds from cached hostfile

 * clearos: clearos.uberglobalmirror.com

 * clearos-centos-sclo-rh: download1.clearsdn.com

 * clearos-contribs: clearos.uberglobalmirror.com

 * clearos-fast-updates: download1.clearsdn.com

 * clearos-infra: clearos.uberglobalmirror.com

repo id                                              repo name                                                              status

clearos/7                                            ClearOS 7 - x86_64 - OS                                                  649

clearos-centos-sclo-rh/x86_64                        CentOS-7 - x86_64 - CentOS Software Collections                        8,219

clearos-contribs/7                                   ClearOS 7 - x86_64 - Contribs                                            131

clearos-fast-updates/x86_64                          ClearOS 7 - x86_64 - Fast Updates                                          3

clearos-infra/7                                      ClearOS 7 - x86_64 - Infrastructure                                       16

repolist: 9,018

As for the updates / Cloud it shows : Up-to-Date.

Any idea ?

SL

'[Errno -2] Name or service not known' is typically a DNS issue and not a proxy issue. Please ensure that you can get proper IP address resolution to the hostnames required. Your proxy service will need to know the IPs involved. It is not important that you are able to ping them but you should be able to resolve the IP behind a proxy server.

Also, do updates work using the 'Software Update' app in the Cloud Category of the menu system?