Resurrecting Apt Zeroconf
I have been toying with a bunch of apt-proxy-like things since I read jdub’s blog about approx. apt-cacher-ng seemed to be working but tends to wonk out on occasion. I decided to try squid for a bit (because then I could use it to cache other things, not just debs).
So after a while of messing with this stuff I decided “this is dumb, why aren’t the computers just maintaining themselves, I am doing too much work for this crap.” I didn’t want to go computer to computer setting a proxy and I didn’t want to mess around with a transparent proxy so I was just annoyed. (And lazy!)
apt-zeroconf was doing this nice and automatically in the past but it was dead upstream and the website disappeared. Maybe if someone fixed this up this would be a nice little solution for home users with multiple PCs. Ends up that Jeremy Austin-Bardo had written a blueprint and a spec. So I gave him a call and now we have started a project to resurrect apt-zeroconf.
The idea is simple. On your home machines you check a box in the gui someplace, and then all your machines share their apt packages with each other. Which means that on release day you upgrade one machine, and then when you upgrade the others you don’t have to redownload everything again. Nice huh? Combine this with deb delta syncing and you’ve got a nice little bandwidth saving solution.
The project is small so some of the guys from the Michigan LoCo will be looking into this as their summer project. If you know python and avahi we could use your help. Also, something changed in pyinotify that we can’t figure out, so help there would be appreciated. Branch away and feel free to join the team and mailing list for updates.
..it should have a ‘use bit torrent for updates and software installation’ option too.
I am unsure of this. First off, this tool is designed to be on a local network, so having bittorrent in a LAN doesn’t seem like a big win to me. Secondly, torrents are great for big files but I am unsure if sharing tons of small debs is useful either. And hopefully for karmic we’ll have delta debs so that’s a savings as well.
Has anyone actually ever done a measurement of how well packages over bittorrent would work? Or is it just a “that would be cool” thing?
Maybe it would be better to make the system have layers. First it would have apt-zeroconf if it was enabled. If the package wasn’t found, then it would pass off to the next layer. That layer could be configured to be an apt bittorent system or just plain http repositories. Also, aren’t official repository packages hash and signed, so no one could change them without haveing them rejected?
yes!
Apt-zeroconf was great until it broke somehow. It saved me tons of bandwidth and sped up updates by lots. A system like this should be default for Ubuntu, IMHO.
Indeed. We’re shooting for making it work first and having it in the archive. Also right now it has no concept of security as far as we can tell, so things need to be fixed before someone would even propose this for main and default. Still, if we can get this fixed and in universe it would be great.
The only security-related issue I see with it is that you could DOS an apt-zeroconf server by requesting packages over and over again, and it obviously is never a good idea to enable more services from a security standpoint.
I also don’t really know what apt-zeroconf would do to a network with 1000+ ubuntu boxes.
I guess this will never be on by default, but making it available easily with a checkbox or such might be nice.
Current caching solutions are completely inadequate for home users and small businesses, especially when you have laptops in the mix that travel from network to network. I’ve looked into apt-cacher and friends, but dismissed them as soon as I realized that this meant i had to edit my sources.list every time i took my laptop or netbook out of the house and wanted to update it.
I’m certainly willing to help you with any beta/alpha testing you want done.
Hey,
the DOS problem is not a that big issue as you can do this with every network service this way.
A problem though is: imagine a node that announces it got lots of files, for example by requesting the lists of all others and saying it got all of them. And if someone requests any file just send them one byte every minute in an endless loop. Now everyone requesting a file from this node will hang. Solvable by using somethig like bittorrent for the lokal network: Implementing requesting the same file from more than one note and mark slow ones.
The place where security is handled should be apt and dpkg thats the only good way – but checking within apt-zeroconf too for marking bad nodes as plus should be considert.
Another note:
If you want resurrecting apt-zeroconf – did you contact the original authors and talked with them?
thanks. you are a hero
Please put it in Debian too.
Thanks a lot my friend for taking up this crucial issue.
If due to security reasons aptzeroconf fails to make it to repositories, please do release a version which works on karmic (and jaunty if possible), which can be downloaded and installed separately (as we have been doing till now)!
something built into synaptic to do a similar job is however the most preferable option…
All the Best!
I’ve been using Apt-zeroconf with Ubuntu 8.10 for a project with some students during my visit at a university in Rwanda, Africa these last few months and it was a real bandwidth saver. Our connection to the outside world would do 8 KB/s on a good day, so having to download the packages only once really helped a lot. Using apt-zeroconf was the best solution, since we didn’t have control of a server (to set up a proxy), hardware is expensive and the network infrastructure (both inside the university and outside) is flaky and changing constantly.
Delta debs will be just as much and maybe even more of a bandwidth saver. Looking forward to both!
I LIKE!
Considering I’ve got 2 Ubuntu boxes at home, this would be good.
When you add in the 3 Ubuntu boxes I personally manage at work, plus the 4 other installs of Ubuntu that I know of at my job, this becomes a great bandwidth saver.
I noticed you mentioned having this on a GUI… I hope the project isn’t tied to X. Imagine setting up Ubuntu Server with apt-cache, serving up its cached packages to GUI clients with apt-zeroconf. Or imagine if you’ve got a serverfarm, using the two to keep all of the servers updated with minimal bandwidth usage.
I always use apt-zeroconf, and it works well. I just installed in Karmic it from my apt-zeroconf_0.4.1_all.deb.
At first it did not work but it turns out something changed in pyinotify, but it can be “fixed” by editing the apt.py file, look for the pyinotify.EventsCodes.IN_MOVED_TO
around line 53 and change it to
pyinotify.IN_MOVED_TO
Do this for the other 3 lines, and restart apt-zeroconf. As always, make your checks, I might end up blowing up your computer
Yeah this is appropriate for small business and homes, if someone has 1000+ ubuntu boxes then clearly a dedicated mirror is a better solution.
A caching proxy is allways nice. There was some code around so apt-zeroconf autodedects apt-cacher proxys and prefer them over direct conncetion. The main usecase for apt-zeroconf are laptops that are not allways in the same network so a static proxy does not work.
So even in big networks if you got mobile devices it could be usefull.