Prefix/Cygwin/X

This article describes how to Article description::get Portage to build usable Qt/Gtk-apps on Cygwin.

Preface
Before reading this, you should be familiar with "Gentoo Prefix" and "Prefix on Cygwin".

It would be useful, if you already set up a Cygwin/X system and bootstrapped a Gentoo Prefix on it. However, reading about the theory might be enough, as you'd have to throw it away later anyway. ;-)

Cygwin/X
Before thinking about bootstrapping, a working Cygwin/X system should be at hand. This is not as trivial as one might think. For best results some things should run as daemons, like they do on any GNU/Linux system. But as you are on Windows, this means you have to set up some of the Daemons as Windows Services.

Don't panic, that one is easier than you think.

Basic installation
I am using the cygwin setup.exe directly using a windows command prompt. That is quicker than searching&clicking every package together.

The following commands can be used to install a Cygwin/X system, that already enables Audio, adds a tray icon, dbus support, Kerberos ticketing, private key management, postfix setup, ssh access, system message logging and an xdg-menu-icon.

I bet you didn't know half of that was possible, right? Me neither until a couple of weeks ago...

Important: The *-devel packages are chosen so the later Gentoo prefix does not need concurrent packages to what is running anyway, or what is almost impossible to build on cygwin using portage. We'll tell portage via package.provided what's what.

You can combine the package lists, of course, and install everything at once.

Basic system
Install a rudimentary system that let's you start an X-Server and bootstrap Gentoo:

Switching the user context without password
Daemons must be able to switch user context. Either to gain special privileges or to drop unneeded privileges.

For the full details please see the corresponding part in the cygwin ntsec guide.

For the daemons you see above, we need two methods. The first is to configure cyglsa, the second is to add some dummy users to Windows.

Enabling LSA authentication
Start a Cygwin64 Terminal with Administrator privileges and issue the following command:

But before rebooting the machine, you can add the dummy users mentioned above.

Adding dummy users
Actually this is only one group and two users.

If you want to use postfix, do the following:


 * 1) Right click on the Windows start button.
 * 2) Select "Computer Management"
 * 3) Expand "Local Users and Groups"
 * 4) Select "Users"
 * 5) Add a new user named "postfix", any password, with "User cannot change password" and "Password never expires" checked. Make it a member of your general "Users" group.
 * 6) Select "Groups"
 * 7) Add a new group named "postdrop"
 * 8) Add "postfix" and your local user to the group.

At least you need the "nobody" user:


 * 1) Repeat Steps 1-4 above
 * 2) Add a new user named "nobody", any password, with "User cannot change password" and "Password never expires" checked. Make it a member of your general "Users" group.

That's it. The installation of openssh daemon will add a sshd user, and cygserver will add a cyg_server user. Everything else will use the SYSTEM account.

Making the daemons work
In order for the system daemons to work, Windows must be told how to start them, and when. Here are the steps needed for the various daemons. Please make sure that LSA authentication is installed (see above) and Windows was rebooted. Further the dummy users mentioned above should be set up already.

Cygwin Server
This is needed for IPC message queues, semaphores and shared memory support. More information is available on the Cygserver page.


 * 1)  Just follow the on-screen information.
 * 2) Go into the windows services panel and search for "CYGWIN cygserver".
 * 3) In its properties Change "Startup type" to "Automatic (Delayed Start)". The normal automatic start may be too fast. If it is, you get weird UID->Name resolutions.

dbus system bus
It is good to have a system bus. Most applications run fine with a session bus only, but a system bus makes live much easier.


 * 1)  Just follow the on-screen information
 * 2) Go into the windows services panel and search for "CYGWIN D-Bus system service".
 * 3) In its properties Change "Startup type" to "Automatic (Delayed Start)".
 * 4) Open a command (CMD) prompt with administrator privileges and add cygserver to the dependencies: The  is a keyword, there really must be a space between it and the dependency list!

syslog-ng system logger
With this you get back.


 * 1)  Just follow the on-screen information.
 * 2) Go into the windows services panel and search for "CYGWIN syslog-ng".
 * 3) In its properties Change "Startup type" to "Automatic (Delayed Start)".
 * 4) Open a command (CMD) prompt with administrator privileges and add cygserver to the dependencies: The  is a keyword, there really must be a space between it and the dependency list!

Avahi Daemon
Well... yeah... You must decide whether you need this, or not...


 * 1)  Just follow the on-screen information.
 * 2) Go into the windows services panel and search for "CYGWIN Avahi service".
 * 3) In its properties Change "Startup type" to "Automatic (Delayed Start)".
 * 4) Open a command (CMD) prompt with administrator privileges and add dbus to the dependencies: The  is a key... well... you got it, right?

Postfix Mail service
If you want to send mails from the console, or any application that uses postfix, you must install its windows service.


 * 1)  Just follow the on-screen information.
 * 2) Go into the windows services panel and search for "CYGWIN postfix".
 * 3) In its properties Change "Startup type" to "Automatic (Delayed Start)".
 * 4) Open a command (CMD) prompt with administrator privileges and add syslog-ng to the dependencies:

ssh daemon
If you want to use SSH to log into your cygwin, then please, go ahead! :-)


 * 1)  Just follow the on-screen information.
 * 2) Go into the windows services panel and search for "CYGWIN sshd".
 * 3) In its properties Change "Startup type" to "Automatic (Delayed Start)".
 * 4) Open a command (CMD) prompt with administrator privileges and add syslog-ng to the dependencies:

newlib-cygwin - compiling yourself
In the Prefix on Cygwin guide you have read that you need a cygwin1.dll with the "forkables" patches applied.

It should be sufficient to use the pre-compiled one, at least as far as I know. However, if you want to compile your own cygwin1.dll, then this is the section for you.

Needed packages
To be able to successfully configure and compile the project, you will need a few extra packages:

Clone the tree
Which tree you use is up to you and your discretion

These are what the pre-compiled cygwin1.dll mentioned above is made off. These are the same Forkables and Gentoo patches, but rebased on the official master repository, thus including all official fixes.
 * https://github.com/haubi/newlib-cygwin.git (Branch: Gentoo)
 * https://github.com/Yamakuzure/newlib-cygwin.git (Branch: master)

If you need advice, go with the first. I can not guarantee, that my sources do more than compile cleanly, as I do not have the time to check whether all rebased patches are still in whole, and whether the automatic resolving of any conflicts hasn't caused any damage.

Configure and build
As you now have all the packages you need, you can configure and build the whole tree.

Whether you put in these FLAGS or not is up to you. it should just work without them. But --prefix and --build are important.

When the configure stage is finished, you can build everything. It is best not to push too hard against the windows kernel. The following is well tested on Windows 10 with an 8-core-CPU:

Install
Yes, you read right. Install this. But not in the root folder, that wouldn't work, because cygwin1.dll is in use. Install it in an image path like this:

The true installation can only happen if absolutely no cygwin process is running. So quit the xdg-menu, close all consoles/windows, and stop all Cygwin services. On Windows 10 you can do the latter using the Task Manager, it has a Services tab. Under Processes you might want to check whether there is no dangling dbus process left behind.

Once everything is stopped, you can use the Windows Explorer or any other file manager to copy the contents of your image directory to your cygwin root directory.

Re-Basing DLLs (fix fork failures)
The dreaded fork failure can happen from time to time. There is much potential for windows software to interfere with Cygwin. Please read "Fixing forkfailures" in the Cygwin FAQ for general information.

However, triggering a full rebase by the Cygwin setup program is meaningless for Gentoo, we are somewhere else. The solution is simple, we do it ourselves.

Find all relevant libraries
There is more to this than just re-basing. There is also the possibility, that a DLL lacks the executable bit. But DLLs must be executable, or it can not be used. Static libraries on the other hand, do not need that bit set.

I have written this little helper script to beat both issues with one bat. Create a list of DLLs to rebase, and fix the executable bit where appropriate. Just adapt the setting to your need.

This will re-create, which lists all possible re-basable libraries.

Do the rebase
The rebase is done in these simple steps:


 * 1) Close all Cygwin processes and log out of your Cygwin X session.
 * 2) End all running cygwin services. The Task Manager in Windows 10 has a "Services" tab you can use.
 * 3) Start a Command Prompt (CMD) with administrator privileges.
 * 4) If the above command tells you that only dash.exe must be running when rebasing, you have some Cygwin processes left. You can kill them (bash.exe, dbus, etc.) using the Task Manager.
 * 1) If the above command tells you that only dash.exe must be running when rebasing, you have some Cygwin processes left. You can kill them (bash.exe, dbus, etc.) using the Task Manager.

You are done. Start all Cygwin Services, you X Session, open a console and continue with your bootstrapping.

Bootstrapping the Prefix
Just do the bootstrap like described on the Project:Prefix/Bootstrap page, unless you do want to start of with some pre-configuration. I prefer the EPREFIX to be "/gentoo", and will refer to it in this way.

If you'd like to use the latest portage tree, you can tell the bootstrap script to do so by issuing: However, I have already had fails due to that download being not what I wanted. It really downloads, which might or might not work. The prefix tree, however, will work.

Things to note down
In case of failure, you can often use the command to finish a merge. But when you are outside the bootstrap script and back on your terminal, the environment is back to normal.

When you have told the script what your EPREFIX should be, it'll print the important environment variables for you. You may want to copy them somewhere safe and use them to prefix a later possible manual run.

These are what I have on my machine: EPREFIX=/gentoo CHOST=x86_64-pc-cygwin PATH=/gentoo/usr/bin:/gentoo/bin:/gentoo/tmp/usr/bin:/gentoo/tmp/bin:/gentoo/tmp/usr/local/bin:/usr/bin:/bin:/cygdrive/c/Windows/System32:/cygdrive/c/Windows MAKEOPTS=-j5

Custom ebuild script for fixing issues
If an ebuild does not finish merging, like gcc always halting at the end of the compile stage, at least on my machine, it may be useful to have a custom ebuild script that takes care of the above environment values.

The following is such a script, that I have in ~/sys/bin, and it works pretty well. At least for me. Adapt and use it if you like.

Possible Preparations
There are a few things you can do before bootstrapping your gentoo prefix. Not much, really, and everything here is purely optional.

Build in RAM
If you have enough RAM, you can have portage do the builds there. Windows does not know tmpfs, but there is a nice freeware tool you can use to setup RAM disks. ImDisk Virtual Disk Driver for Windows can be used to set up a RAM DISK and mount it in /build, or wherever you want. Just do not forget to tell portage about it by adding export PORTAGE_TMPDIR=/build (or wherever you mount your RAM Disk) to your ~/.bashrc. Alternatively you can set up a make.conf to start with, and add the PORTAGE_TMPDIR entry there.

Pre-Setup a make.conf
I'll post here my minimum make.conf for bootstrapping. It adds -march=native to the C[XX]FLAGS, sets CPU_FLAGS_X86 and enables split logs in /gentoo/var/log/portage.

If you really want to use this, put it in /gentoo/etc/portage and be damn sure to check CPU_FLAGS_X86 against your CPU!

Portage tree too old
If any of the problems below signals that your portage tree is too old, either because using didn't do what you expected, or because your  script is too old, you do not have to start over. Just fix this and continue.


 * 1) Download the latest boostrap-prefix.sh script.
 * 2) Restart the script, but kill it with Ctrl+C after it has synced in the latest prefix tree, and starts to continue the bootstrap.
 * 3) If you have binutils-2.28.1 installed, do the following steps:
 * 4) Source  and
 * 5) If you have gcc-5.3.0 installed, do the following steps:
 * 6) Make sure that the following command shows that gcc-6.4.0 is selected:
 * 7) Source  and
 * 1) If you have gcc-5.3.0 installed, do the following steps:
 * 2) Make sure that the following command shows that gcc-6.4.0 is selected:
 * 3) Source  and
 * 1) Source  and

After this you can continue your bootstrap process.

Problems, Breaks, Failures
There are some things that might fail. Here are the common ones for me.

libffi: Bootstrap dies
This is serious. It means that is not working as it should. If you have used my variant of the newlib from github.com/Yamakuzure, then please let me know. It means that the application of the forkable patches went wrong!

gcc: Never ending merge
On some rare occasions the merge of sys-devel/gcc will halt after the compilation stage is over.

Luckily this is easy to fix, as long as you noted down the values mentioned above, that were displayed by the bootstrap script. Just substitute the values here with your values.

Or even better: Use my script above, it'll make things easy for you. ;-)

Fix in Stage 2
In Stage 2 use the fixing script above like or issue: $ EPREFIX=/gentoo/tmp \ CHOST=x86_64-pc-cygwin \ PATH=/gentoo/usr/bin:/gentoo/bin:/gentoo/tmp/usr/bin:/gentoo/tmp/bin:/gentoo/tmp/usr/local/bin:/usr/bin:/bin:/cygdrive/c/Windows/System32:/cygdrive/c/windows \ /gentoo/tmp/usr/bin/ebuild /gentoo/usr/portage/sys-devel/gcc/gcc-6.4.0.ebuild merge to finish the merge.

Afterwards you should see: * Switching native-compiler to x86_64-pc-cygwin-6.4.0 ... * Updating LTO plugin symlink in /./gentoo/tmp/usr/x86_64-pc-cygwin/binutils-bin/lib/bfd-plugins    [ ok ]

As this ends stage 2, you can do like portage suggests, before restarting the bootstrap. $ . /gentoo/tmp/etc/profile $ . ~/.bashrc

Fix in Stage 3
If you use the default, or if you changed that but exported the value in your .bashrc, you can proceed like in Stage 2.

Otherwise, if you changed in your make.conf, but did not export the value, you have to fix the build like in Stage 2, but set  to /gentoo/var/tmp like this:

In Stage 3 use the fixing script above like or issue: $ EPREFIX=/gentoo \ CHOST=x86_64-pc-cygwin \ PATH=/gentoo/usr/bin:/gentoo/bin:/gentoo/tmp/usr/bin:/gentoo/tmp/bin:/gentoo/tmp/usr/local/bin:/usr/bin:/bin:/cygdrive/c/Windows/System32:/cygdrive/c/Windows \ PORTAGE_TMPDIR=/gentoo/var/tmp \ /gentoo/tmp/usr/bin/ebuild /gentoo/usr/portage/sys-devel/gcc/gcc-6.4.0.ebuild merge and it will (hopefully) be finished.

This always did the trick for me. This halt happened for me in stage2 and stage3. Well, at least it did not occur in any other situation for me.

After the merge finishes, you should see something like: * Switching native-compiler to x86_64-pc-cygwin-6.4.0 ... [ ok ]

After that you can source and your  and just restart the bootstrap script. It will continue where it left off.

app-arch/xz-utils fail in Stage 3
In stage 2, "" merges just fine, using from binutils-2.29.1 provided by cygwin. But in stage 3 the provided by the available binutils-2.28.1 breaks. It does not recognize the parameter in its call from libtool.

To fix this we are going to force the xz-utils build system to use the cygwin version of.

~ $ EXTRA_ECONF="ac_cv_prog_RC=/usr/bin/windres.exe" ~/sys/bin/do_ebuild /gentoo/usr/portage/app-arch/xz-utils/xz-utils-5.2.3.ebuild clean configure (...) checking for x86_64-pc-cygwin-windres... (cached) /usr/bin/windres.exe (...)

This looks good, so call "" instead of "" to install a working xz-utils. This time it should work.

Afterwards just restart the bootstrap script, it should continue where it left off.

sys-apps/coreutils-8.29 fails in Stage 3
As of writing this, fails with the message: mv: cannot stat '/build/portage/sys-apps/coreutils-8.29/image/gentoo//usr/libexec/coreutils/libstdbuf.dll.a.exe': No such file or directory

Unfortunately the patches and methods to emerge coreutils-8.28 do not work with 8.29 and the only way to get around this, right now, is to use to merge the older version.

Further you need to mask the new coreutils version, at least for now. $ mkdir -p /gentoo/etc/portage/package.mask $ echo ">=sys-apps/coreutils-8.29" >> /gentoo/etc/portage/package.mask/coreutils_limit

child_info_fork::abort: : Loaded to different address
The re-basing did not work as it should have. Before continuing, you should rebase everything.