Updates are working again !!

Richard George wrote:

Check the full log - the sorting order in the displayed list is screwed (I reported this on Nick's Packages being updated to the Community tonight (16/07/2019) post.

Sorting order looks fine at my system
The yum.log also listed no updates anymore from 17-July

Check the full log - the sorting order in the displayed list is screwed (I reported this on Nick's Packages being updated to the Community tonight (16/07/2019) post.

I've mentioned it to Dave, thanks.

Hi,

It looks that clearsdn-antimalware and clearsdn-antispam doesn't update anymore since 17/7.

Nick Howitt wrote:

@blue presto,
Did you see any errors when you tried the update? Also can you have a look in /var/log/system for the automatic updates overnight?

If you would like Clearcenter to investigate, please raise a General Enquiry ticket at https://secure.clearcenter.com/portal/ticket_manage.jsp and submit remote login credentials to https://secure.clearcenter.com/portal/system_password.jsp. Obviously make sure you have a strong password and open the incoming ssh firewall. Alternatively, if you want to use ssh keys only (rather than user/password), please indicate on the ticket and Clearcenter will respond with their public keys.

No errors. Just says "no update available" similar to what was happening with the other packages mentioned earlier in this thread before the server side fix they implemented.

@blue presto,
Did you see any errors when you tried the update? Also can you have a look in /var/log/system for the automatic updates overnight?

If you would like Clearcenter to investigate, please raise a General Enquiry ticket at https://secure.clearcenter.com/portal/ticket_manage.jsp and submit remote login credentials to https://secure.clearcenter.com/portal/system_password.jsp. Obviously make sure you have a strong password and open the incoming ssh firewall. Alternatively, if you want to use ssh keys only (rather than user/password), please indicate on the ticket and Clearcenter will respond with their public keys.

Dave Loper wrote:

Hmmm...it is possible that you were in the update hole. That is what I'm calling it anyways. The issue stems from the way that we process updates for automatic release. Because of the way that upstream designates their release folders (ie. 7.5.1804, 7.6.1810) the repo location for upstream packages has to change. We manage this in the app-base package. This is why the manual instructions are to run:

yum update app-base

yum update

What this means is that if you get an app-base, you won't get the next version until the next night. This is why the kernel was semi-pushed when the release happened and had to be withdrawn (some are using it, some aren't). In the future, we will be watching the updates and if app-base is set for update, it will process through a second time.

It is possible that the updates were not processing while it was resolving the two-step delay. I'm speculating here. We couldn't replicate it normally. Let us know if it fails again. We are transitioning the way these updates get distributed so that it follows the yum model more fully.

I tried those as well. IDS still stuck on Feb 28

Thanks Dave

Hmmm...it is possible that you were in the update hole. That is what I'm calling it anyways. The issue stems from the way that we process updates for automatic release. Because of the way that upstream designates their release folders (ie. 7.5.1804, 7.6.1810) the repo location for upstream packages has to change. We manage this in the app-base package. This is why the manual instructions are to run:

yum update app-base

yum update

What this means is that if you get an app-base, you won't get the next version until the next night. This is why the kernel was semi-pushed when the release happened and had to be withdrawn (some are using it, some aren't). In the future, we will be watching the updates and if app-base is set for update, it will process through a second time.

It is possible that the updates were not processing while it was resolving the two-step delay. I'm speculating here. We couldn't replicate it normally. Let us know if it fails again. We are transitioning the way these updates get distributed so that it follows the yum model more fully.

Hey Dave,

I'm replicating your setup in a lab. What version of ClearOS did you install and what level is your server at now?

ClearOS 7.5.0 (Final) Business Edition
3.10.0-862.6.3.v7.x86_64 Kernel Version

They’re showing updated now, so maybe there was a delay after the new install?

Dirk,

I'm replicating your setup in a lab. What version of ClearOS did you install and what level is your server at now?

I actually reloaded the machine with ClearOS maybe a day or so after it happened, but I didn't actually take it online and login to the Webconfig until this morning. When I did, I went through the Wizard and registered the system (along with my ClearSDN credentials), then chose the different apps to install, then updated everything from the Webconfig. Everything has updated from what I can tell, except for my ClearSDN subscriptions, i.e. the different signature updates. I've even got my website back up. The only difference between then and now is I was using it as a gateway, but now just a web and ftp server in standalone mode.

I really don't know what is happening on this one and will need to escalate it. How long ago was it that you reloaded your machine?

I wonder if it would help uninstalling the Antimalware Rules and Antispam Rules and reinstalling them.

Can you you try a:

yum clean all && yum update clearsdn*

Same result - no packages marked for update

Can you you try a:

yum clean all && yum update clearsdn*

What happens if you do it manually?

# yum update clearsdn*

Loaded plugins: clearcenter-marketplace, fastestmirror

ClearCenter Marketplace: fetching repositories...

Loading mirror speeds from cached hostfile

 * clearos: mirror2-newyork.clearos.com

 * clearos-centos-sclo-rh: download1.clearsdn.com

 * clearos-centos-verified: mirror2-newyork.clearos.com

 * clearos-contribs: mirror2-newyork.clearos.com

 * clearos-epel-verified: mirror2-newyork.clearos.com

 * clearos-fast-updates: download1.clearsdn.com

 * clearos-infra: mirror2-newyork.clearos.com

 * clearos-verified: mirror2-newyork.clearos.com

 * private-clearcenter-ad: download1.clearsdn.com:80

 * private-clearcenter-antimalware: download2.clearsdn.com:80

 * private-clearcenter-antispam: download1.clearsdn.com:80

 * private-clearcenter-business: download3.clearsdn.com:80

 * private-clearcenter-dnsthingy: download1.clearsdn.com:80

 * private-clearcenter-dyndns: download3.clearsdn.com:80

 * private-clearcenter-ids: download1.clearsdn.com:80

 * private-clearcenter-kopano-basic: download3.clearsdn.com:80

 * private-clearcenter-master-slave: download4.clearsdn.com:80

 * private-clearcenter-plex: download2.clearsdn.com:80

 * private-clearcenter-rbs: download3.clearsdn.com:80

 * private-clearcenter-verified-updates: download1.clearsdn.com:80

No packages marked for update

My ClearOS gateway crashed a little more than a week ago. I finally got it back up and running. Turns out my LAN nic went bad, so now I just have it in standalone mode to use for a web server and an ftp server. I'm considering Nextcloud. I have a Squarespace website that I embed videos in that are stored on my ClearOS box via a Joomla site in my webroot. Plus I have a file manager (via my Joomla site) that customers use. I'm really tired of being at the mercy of remote server providers for these needs, although there's something to be said for backups should your server go down. All this to say, since I just got my ClearOS box back up in standalone mode (with firewall), maybe this has something to do with the lag in antimalware signature updates? My A record is updated in ClearSDN and I can access my Webconfig.

Nick Howitt wrote:

IDS updates have always been sporadic, between one and four times a month. These are the updates I've had thin year:

Jan 09 03:21:49 Updated: clearsdn-intrusion-protection-6.1-20190108.1555.noarch

Jan 18 06:02:51 Updated: clearsdn-intrusion-protection-6.1-20190116.1231.noarch

Jan 29 06:46:46 Updated: clearsdn-intrusion-protection-6.1-20190128.1343.noarch

Feb 09 04:16:03 Updated: clearsdn-intrusion-protection-6.1-20190208.1445.noarch

Feb 28 02:44:03 Updated: clearsdn-intrusion-protection-6.1-20190227.1036.noarch

Apr 05 06:31:17 Updated: clearsdn-intrusion-protection-6.1-20190403.1428.noarch

What shows in your yum.log file?

My last update was the Feb 28th one. I never got the Apr 05. I was hoping this one would re-start along with the others but it never did.
Tried yum update clearsdn* and "no packages marked for update"

There was a pause but they have restarted:

grep clearsdn /var/log/yum.log

<snip>

Mar 26 16:11:53 Updated: clearsdn-antispam-6.1-20190325.1025.noarch

Mar 26 16:12:40 Updated: clearsdn-antimalware-6.1-20190325.1025.noarch

Apr 03 03:45:29 Updated: clearsdn-antispam-6.1-20190402.2021.noarch

Apr 04 01:01:48 Updated: clearsdn-antimalware-6.1-20190403.1025.noarch

Apr 04 01:02:36 Updated: clearsdn-antispam-6.1-20190403.1025.noarch

Apr 05 01:42:26 Updated: clearsdn-antispam-6.1-20190404.1025.noarch

Apr 05 01:44:23 Updated: clearsdn-antimalware-6.1-20190404.1025.noarch

Apr 05 06:31:17 Updated: clearsdn-intrusion-protection-6.1-20190403.1428.noarch

Apr 06 02:15:26 Updated: clearsdn-antimalware-6.1-20190405.1025.noarch

Apr 06 04:36:51 Updated: clearsdn-antispam-6.1-20190405.1025.noarch

Apr 07 02:51:36 Updated: clearsdn-antispam-6.1-20190406.1025.noarch

Apr 07 03:38:27 Updated: clearsdn-antimalware-6.1-20190406.1025.noarch

Apr 08 01:10:25 Updated: clearsdn-antimalware-6.1-20190407.1025.noarch

Apr 08 02:54:15 Updated: clearsdn-antispam-6.1-20190407.1025.noarch

Apr 09 04:30:28 Updated: clearsdn-antimalware-6.1-20190408.1025.noarch

Apr 09 05:41:27 Updated: clearsdn-antispam-6.1-20190408.1025.noarch

Apr 10 03:32:05 Updated: clearsdn-antimalware-6.1-20190409.1025.noarch

Apr 10 03:32:55 Updated: clearsdn-antispam-6.1-20190409.1025.noarch

Apr 11 01:33:51 Updated: clearsdn-antimalware-6.1-20190410.1025.noarch

Apr 11 01:34:45 Updated: clearsdn-antispam-6.1-20190410.1025.noarch

Apr 12 04:46:58 Updated: clearsdn-antispam-6.1-20190411.1025.noarch

Apr 12 06:06:30 Updated: clearsdn-antimalware-6.1-20190411.1025.noarch

What happens if you do it manually?

yum update clearsdn*

So Antimalware and Antispam signatures have not been updated since March 27, 2019, when they used to be daily. It's been a minute. What's up fellas?

IDS updates have always been sporadic, between one and four times a month. These are the updates I've had thin year:

Jan 09 03:21:49 Updated: clearsdn-intrusion-protection-6.1-20190108.1555.noarch

Jan 18 06:02:51 Updated: clearsdn-intrusion-protection-6.1-20190116.1231.noarch

Jan 29 06:46:46 Updated: clearsdn-intrusion-protection-6.1-20190128.1343.noarch

Feb 09 04:16:03 Updated: clearsdn-intrusion-protection-6.1-20190208.1445.noarch

Feb 28 02:44:03 Updated: clearsdn-intrusion-protection-6.1-20190227.1036.noarch

Apr 05 06:31:17 Updated: clearsdn-intrusion-protection-6.1-20190403.1428.noarch

What shows in your yum.log file?

We were told they fixed something server side and now the updates are working again! Except for IDS. IDS is connected OK to download3.clearsdn.com:80
Is anyone else having this issue?

Currently the downloads is in a redundant mirror mode while we are in transition, if you ping the download[1-4] they will all go to the same place for now and the master mirror is not exposed, this will morph again to the distributed mirror model after the dust settles. We will make more progress on this tomorrow morning after our internal meeting.

To that end we want to onboard two processes that we have talked about for a while,

1) We want to move the process of passing updates from test to ClearOS through a community driven approval process.
2) We want to implement a community-based agenda and directives for the ClearOS project moving forward. This will be a meeting that will be open to everyone worldwide. The time that I suggest is to place this in such a way that the Pacific regions (lowest population) are in the middle of the night. (Sorry NZ you'll need to stay up late). For me this means waking up before 6 AM, 8AM for Eastern US, but evening for our Indonesian friends. I suggest a Saturday for this purpose.

Perhaps a different thread to talk about this would be best.

Thanks Dave - manual intervention to band-aid without fixing the root cause often leads to unintended consequences. Trust the new sync process will render this type of activity unnecessary and save you time... Will sit tight in the meantime.

For some reason not all of my machines decide a one particular mirror as being the closest. That's interesting in that sometimes update(s) on some machines appear one, two or occasionally three days before others. All running the same 7.x and running yum at the same time of day. ** Will be interesting to see whether the sync process changes this, or some mirrors are just slow to get updated. Similarly this problem with the xml files didn't show on all machines.

** run "yum update -y --downloadonly" daily on all machines sending me the output in emails. This gives me warning of any updates. Then decide when to manually install which updates for each machine. Auto update is turned off - too problematic. Therefore any inconsistency between whatever mirrors used by the machines here is easily detected...

I know the repodata that you are getting and why it is too old, we'll have the problem sorted in a bit. The data is the same but the new sync process pushed some older xml files over the top of the ones we had to manually generate in our old sync process. We are getting that sorted, sorry for the kerfuffle.

We also are working on the kopano_basic issue. Thanks for your patience.

Sorry - 7.x
Sure - knew already that would stop the message - but doesn't solve the real problem...
Would only result in me downloading xml stuff that is way too old... the mirrors need fixing... the source of the problem

The automatic updates of antiX packages should be working now and we are digging into the repo xml issues.

Hey Dave, I'm not seeing any updates available for antiX subscriptions. A "yum clean all" seems to have rid me of my kopano_basic mirror errors, plus "Trying" no longer shows up in the Webconfig; just "The latest software updates are installed."

How often do the IDS Signatures get updated? I haven't seen one since the end of February.

Sorry for the mess ladies and gentlemen, we are in the process of migrating the 'downloads' mirrors to different hosting solutions. The automatic updates of antiX packages should be working now and we are digging into the repo xml issues. Thanks for reporting the issues.

@Dirk. I am not sure of yours. You are on 7.x, aren't you? I'll ping Dave about it.

Yes I am, Nick.

@Tony, I pinged Dave over the weekend about that when I bumped into it on ClearOS 6. He is looking at it, but the main person will only be around tomorrow. You can get rid of it by doing a "yum clean all". Is your 6.x or 7.x where you are seeing it.

@Dirk. I am not sure of yours. You are on 7.x, aren't you? I'll ping Dave about it.

This isn't good either - just started happening...

Not using downloaded clearos-fast-updates/repomd.xml because it is older than what we have:

  Current   : Thu Mar  7 02:34:49 2019

  Downloaded: Thu Jan 17 00:34:40 2019

clearos-infra                                                                                                        | 3.5 kB  00:00:00     

clearos-updates                                                                                                      | 3.5 kB  00:00:00     

private-clearcenter-dyndns                                                                                           | 2.9 kB  00:00:00     

Not using downloaded private-clearcenter-dyndns/repomd.xml because it is older than what we have:

  Current   : Thu Mar  7 02:34:32 2019

  Downloaded: Tue Aug  8 04:50:57 2017

What's the dealio guys? I'm still getting "no packages marked for update" when trying from shell prompt. Thusly I'm getting "Trying" under available updates in the Webconfig. I'm also getting "HTTP Error 403 - Forbidden" errors for all mirrors for kopano_basic.

Yes, I know. I've just been talking about it with Dave. There has been a massive infrastructure change and this has not quite gone according to plan.

Dave Loper wrote:

This should be fixed by tomorrow.

No updates again. It was working only for 1 day

Dave Loper wrote:

This should be fixed by tomorrow.

yep... they are back !!

This should be fixed by tomorrow.

I had to regenerate all my rpm database files a few weeks ago to increase the size of the database because "rpm -qa" queries were taking too long. Anyway most of my files are dated today as I did a localinstall of a package earlier this morning.

The updates you've listed mainly came from clearos-updates.

Hey Nick, that's what I figured, but I know that ClearOS-Updates was enabled because that's the first one I looked for. Problem is I can't remember which one I had to enable. Wish I had written it down, but I'd swear it was the plain ClearOS repo.

Can you verify if the files in /var/lib/rpm get updated whenever there's an update installed? If you look at the link I shared of an older post, at that time I was getting a repo error in the system log that was dated the same as the files in /var/lib/rpm. I'm not getting one now, but that doesn't mean it's not related.

Hi Dirk,
The updates you've listed mainly came from clearos-updates. This is not normally available for paid users. I'm pretty certain there is an issue with the clearsdn updates as I am seeing noting either. I am waiting to hear back from Dave on them.

Check to make sure your ClearOS repositories are enabled. I can't for the life of me remember which one was disabled, that when re-enabling allowed me to get the latest update. I was noticing the same thing. I want to say it was the ClearOS repository itself, being that's related to pertinent app updates rather than contributors to ClearOS. I don't know how it got disabled. I'm telling you, something got whacked out when files got updated in /var/lib/rpm back in February. Check out this post and you'll see what I mean. I had to rebuild the rpmdb to get it back to working. I never updated the rpmdb that would've caused the glitch.

The following are the latest updates I got after enabling the repository. Sorry I can't remember which repository it was, but it would've been a number of days after the rpmdb rebuild I mentioned above. Dunno if they're linked. Pretty sure it was just the ClearOS repository that was disabled.

libcomps-0.1.8-7.el7 Updated Mar 21, 17:00:25
python2-libcomps-0.1.8-7.el7 Updated Mar 21, 17:00:25
app-firewall-2.6.5-1.v7 Updated Mar 21, 17:00:25
app-attack-detector-2.3.6-1.v7 Updated Mar 21, 17:00:25
app-events-2.5.2-1.v7 Updated Mar 21, 17:00:25
app-firewall-core-2.6.5-1.v7 Updated Mar 21, 17:00:24
app-network-core-2.6.0-4.v7 Updated Mar 21, 17:00:23
app-network-2.6.0-4.v7 Updated Mar 21, 17:00:23
app-attack-detector-core-2.3.6-1.v7 Updated Mar 21, 17:00:23
app-events-core-2.5.2-1.v7

BTW Patrick, what you're noticing was the exact thing I was noticing, since I subscribe to those updates. See if you're getting any repository errors in your system log. Look at the files in /var/lib/rpm for newer dates on them. Read the link I shared above. There's a link inside of that post I shared that steps you through rebuilding the rpmdb if you find it's an issue. Maybe then look at which repositories are enabled after doing the rebuild?

Update: Maybe all I wrote above has nothing to do with your problems. I noticed my subscription updates were last updated March 8. Heck. my IDS Signatures were last updated February 27. I did pay my bill, right? I noticed that the files in /var/lib/rpm are now dated 3/21/19. That's pretty new, but I'm not seeing any errors in the system log or mariadb. That doesn't mean it's not an rpmdb issue again. Do those get updated any time there's an app update installed? Look at the dates of the updates I listed above. They're the same as the files in /var/lib/rpm.