John Marino’s DPorts project, mentioned here briefly before, is interesting. I had two separate people ask me how it works, so a better explanation is in order. I’ve tried it out on a test machine over the past few weeks.
Background:
Dports is an effort to use FreeBSD’s ports system as a base for DragonFly, and the pkg tool as a way to manage binary packages built from DPorts. This is complicated, so I’ll explain each part in order.
- FreeBSD ports are a FreeBSD-specific collection of software installation files that automate building 3rd-party software on FreeBSD. You’ve probably already heard of them. (Note there’s no mention of DragonFly.)
- DPorts is a collection of files that map to existing FreeBSD ports, and contain any changes necessary to make that port also build on DragonFly. Many of those programs build without changes on DragonFly. DPorts builds from source.
- pkg is used for package management, and is usable on FreeBSD and on DragonFly. The binary packages produced from building with DPorts can be installed from remote locations and managed separately using pkg, so that software upgrades and installation can be performed with binaries only. (It’s much faster that way.)
Every port seen in DPorts is known to build on DragonFly. John Marino adds a port only after it builds successfully, using poudriere as a bulk software tool. Ports are only updated to a newer version when that newer version builds, too, so once something arrives in DPorts, it should never break from being updated at some point in the future.
Installing:
To use DPorts, you need two things:
- DragonFly 3.3 or later, though 3.3 is the most recent right now.
- You need to rename /usr/pkg so that your existing pkgsrc binary programs don’t get accidentally used while working with DPorts, causing confusion. If anything goes wrong with DPorts when you are installing it and you want to go back, remove all the DPorts packages and rename /usr/pkg back to normal.
(Don’t confuse pkg, the management tool, with /usr/pkg, the normal installation directory for pkgsrc. ) For the installation of the base port files:
cd /usr make dports-create-shallow
If you’ve already renamed your /usr/pkg directory, git won’t be in your path any more. You can instead download a tarball and unpack it, which also happens to be possible automatically via that same Makefile.
cd /usr make dports-download
Downloading via git is fastest, so if you do need to use the tarball via make dports-download, build devel/git, delete /usr/dports, and then pull it again with make dports-create-shallow. This all comes from John Marino’s Github site for DPorts.
Managing DPorts
DPorts doesn’t use pkg_info, pkg_add, and the other tools traditionally seen on DragonFly for pkgsrc. Instead, package management is done with pkg. Use pkg info, pkg install, pkg remove, and pkg update to list, install, delete, and upgrade various packages on your system. Packages built from source or downloaded as prebuilt binaries are managed the same way, using these tools.
See some of the other writing about pkg for FreeBSD for details on how it works.
Since DPorts doesn’t update a package until it gets a successful build, and installations are of successfully built binary packages, upgrades with prebuilt packages should always succeed. Since they’re binary, they should be fast. There’s a lot of ‘shoulds’ in this sentence, but these are reasonable suppositions.
What about pkgsrc?
Pkgsrc and DPorts shouldn’t be used at the same time, since one system’s packages may be at different versions but still get picked up during building for the other system. That’s about it for restrictions.
I intend to try building an experimental release of DragonFly with DPorts, to see if all the right packages can be added, but no guarantees. DPorts is brand new and does not yet have a repository for downloading packages, so the normal caveats apply; don’t install it on a mission-critical machine, and be ready to deal with any surprises from using it if you do try it out.
What packages are available?
Browsing the Github repo will show you all listed packages. More complex packages like xorg, openjdk7, and libreoffice install, as does xfce. Parts of KDE 3 and KDE 4 are in there. (I haven’t tried either.) I’m not sure about Gnome, but I don’t think anyone ever is. There’s no vim, but there is emacs.
That’s just what I see at this exact minute. It changes daily as more packages are built. Changes from DragonFly builds are sometimes relevant to the original FreeBSD port, so there’s benefits for everyone here.
What next?
Try it now if it has all the packages you need, or wait for a binary repository to be created to speed things up. Remember, this is a new project, so a willingness to deal with problems and contribute to fixes is necessary.
Sounds pretty cool, pkgsrc always felt non-native on DFly (well maybe because it is) and I catched myself typing make instead of bmake more than once…
FreeBSDs new package management tool pkg seems like a big leap forward, too.
Oh, that’s something I forgot to mention – building DPorts stuff uses make, not bmake, so you’d be typing ‘make install clean’ and so on.
Dports is a great achievement and seems native on DFBSD. Is there any Linux Emulation in Dports. Any how many packages are available there.
faisal – there may be linux packages in DFports but I don’t think they’ve even been tried yet. The linuxulator doesn’t work on x86_64 and that’s what my local test machine is, so I don’t know.
I think the port count is over 12,000, but that’s a first pass of successful builds. The number is probably technically lower because of programs that build successfully but do not run correctly. We’ll find out in the coming weeks.
The port count is ~17300 right now.
All the linux emulation packages are i386 only and they haven’t been attempted yet. They are coming though.
I’m just going to say “over nine thousaaaaaaaand!” from now on whenever someone asks me about package count.
I don’t mean to rain on the parade here, but I rather fail to see the point here. Why would one want FreeBSD’s ports on DFly? I’ve used both fbsd ports and pkgsrc and neither one seems superior over the other — at least from a build-things-from-source point of view (I haven’t used the binary features enough to judge.)
1) More than twice as many packages (9,000 vs ~23,000)
2) The packages are more up to date, it some cases by SEVERAL YEARS (yes, some pkgsrc pkgs are updated as fast or faster than ports, but they are exceptions)
3) The binary package handling is integrated, not an add-on (and much, much better than pkg_radd)
4) No obnoxious quarterly branches
5) Integrated in DragonFly using system mk and make, not bmake
6) I prefer options dialog but this is a matter of taste
I will not even talk about behind-the-scene benefits that most users wouldn’t be aware of. Nor will I talk about which is superior technology because this is a highly biased topic (let’s just say that I don’t believe “the gap” is between pkgsrc and ports is as big as pkgsrc devs say it is)
Frankly, I don’t see why this is a mystery.
I have a commit bit to pkgsrc and it’s been indicated that I can get a commit bit to Ports so I think I have better good insight into both systems.
carlo – The advantage of DPorts for me, at least, is that it fixes a working package in place and doesn’t upgrade until the replacement works. I’ve had release builds fail because a pkgsrc package that previously worked, failed, and the only way to fix that is to run a bulk build over the course of weeks, identify the failing packages, and have someone fix them. This is not always feasible, given the interaction of quarterly pkgsrc releases and biannual DragonFly releases.
It also has a clear binary packaging system and upgrade method in place. There isn’t really one for pkgsrc, or rather there’s a bunch, all of which don’t quite cover all the issues. Pkgin handles only binaries, which don’t always build from quarterly release to quarterly release as noted above. pkg_rolling-replace only builds from source, and requires some fiddling with packages for name changes, and so on. The one canonical way that comes with pkgsrc, ‘bmake replace’, isn’t recommended.
This sort of binary packaging system is in theory possible with pkgsrc too, but John Marino’s going with FreeBSD ports for various reasons, including the overall higher count of packages.
Yes, that is another excellent point that I realize I forgot. pkgsrc is very aggressive with “revbumps” which force dependencies to rebuild. But if you build from source, the package you want fails because the dependency fails because it’s already installed.
Then they came up with that rolling_replace tool (I think of it as a hack) but you can easily find yourself in a situation where you can’t even delete the old package and the whole repository is messed up.
with pkg you also get sqlite as the package database and it’s lightning fast. Many pkg_* tools are built in, like “pkg which” and that is fantastic. You can pass any path to pkg and it will tell you what package installed it. Highly useful. That kind of stuff is why I say “it’s integrated” and not an optional bolt-on. Pkgsrc could do the same, but they need to settle on an official approach and tool and fix the mk files to accomplish it.
Pkgsrc was more designed to only get binaries as evidenced by the very poor experience with updating from source. DPorts is also focused on the binary-only experience, but building from source and updating from source should be less problematic if that is what you want to do.
– I believe the equivalent of pkg_rolling-replace is portmaster which feels no less of a hack. The useful tools seem to be “optional bolt-ons” in both cases
– pkg(8) seems like a welcome addition.
– It sounds like some of your grievances could be solved with an automated build attempt/bugreport whenever there is a commit. I take it many pkgsrc developers only test on NetBSD?
I’m not really sure what to think about the other things you mentioned (and some of them sound a tad on the subjective side) — I’m afraid I don’t have enough experience with DFly to be able to judge by myself.
I think both ports systems could use both a new frontend and a backend (The building can still be done with the make rules, but everything else, starting from the dependency resolution, should be done by something else.) Maybe pkg(8) can fill that slot?
carlo – yes, pkg could in theory work for freebsd ports or for pkgsrc. It requires someone to put in the work, as with anything.
An automated build system for pkgsrc would be nice.
I have little experience with portmaster, but from what I know it’s better than pkg_rolling_replace.
I have brought up the QA issues with commits that break non-NetBSD systems a few times and been shot down every time. The official statement paraphrased: Yes, breakage happens, a lot. Non-NetBSD are expected to follow the original committer and clean up his mess. Testing is not required”.
So basically: if there is any testing, it’s NetBSD only (and probably only a small subset of archs for NetBSD). As seen recently with the webmin fiasco, the guy that causes massive breakage isn’t required to fix any of it. That’s not how I roll — If I broken in excess of 40 packages I would revert my commit with significant embarrassment.
Carlo is correct that theoretically we can set up the same “staging” scheme that I’ve set up for FreeBSD. But why bother?
DragonFly has used pkgsrc for at least 6 years. It’s not like pkgsrc didn’t get a fair evaluation. I have several developer-based gripes but dports is about user-facing merits.
The good news is that users aren’t forced to use either system. Both are supported. It is likely that my support of pkgsrc will reduce to Ada packages and fixing unacceptable regressions but I probably won’t fix too many more leaf packages. My time is better spend on dports.
Ah, I did not realise I was talking with John Marino. I did not mean to imply that the same steps should be done for pkgsrc — just that I figured it might be more work to mirror FBSD’s ports tree than tackle the problems with pkgsrc. But I see that many of the problems are political rather than technical — which I find rather strange, since DFly and NetBSD are both listed as Tier 1 platforms for pkgsrc. I surmised you were on equal footing.
This is a common assumption: surely it must more effective to improve pkgsrc than return to ports after 7 years?
I’ve been committing furiously to pkgsrc for about year. I raised the package count by a couple of thousand in that time. I’ve been working dports for a couple of months, and we’ve just passed 17,5000 ports.
I recognized that ports was not as incompatible for dragonfly as most assume. I thought we could get over 12,000 packages pretty easily and this turned out to be a correct assessment.
Regarding the political issue: At the end of the day, pkgsrc exists to serve NetBSD and most of the developers contribute accordingly. Every now and then a suggestion to spin it off as a separate project (make it neutral, make it easier for others to get a commit bit) comes up but is always shot down. I believe this needs to happen but I am skeptical that it will.
The reaction to support gcc47 on DragonFly proved to me that in reality DragonFly is a second-class citizen. On paper, it’s equal footing, but that is where it ends in my opinion.
>You need to rename /usr/pkg so that your existing pkgsrc binary programs don’t get accidentally used
And there is interesting file /var/db/pkg/auditfile. For example:
freetype2 from pkgsrc has version 2.4.11. After installation /var/db/pkg/auditfile contains next line:
freetype2<2.4.11|http://portaudit.FreeBSD.org/1ae613c3-5728-11e2-9483-14dae938ec40.html|freetype — Multi\
ple vulnerabilities.
If I try install xorg from dports then installation fails. Dports has freetype2 with version 2.4.9_1 :(