Contributed by Nik Clayton <nik@FreeBSD.org>.
Once you have synchronised your local source tree against a particular version of FreeBSD (stable, current and so on) you must then use the source tree to rebuild the system.
Take a backup: I cannot stress highly enough how important it is to take a backup of your system before you do this. While remaking the world is (as long as you follow these instructions) an easy task to do, there will inevitably be times when you make mistakes, or when mistakes made by others in the source tree render your system unbootable.
Make sure you have taken a backup. And have a fixit floppy to hand. I have never needed to use them, and, touch wood, I never will, but it is always better to be safe than sorry.
Subscribe to the right mailing list: The -stable and -current FreeBSD code branches are, by their nature, in development. People that contribute to FreeBSD are human, and mistakes occasionally happen.
Sometimes these mistakes can be quite harmless, just causing your system to print a new diagnostic warning. Or the change may be catastrophic, and render your system unbootable or destroy your filesystems (or worse).
If problems like these occur, a "heads up" is posted to the appropriate mailing list, explaining the nature of the problem and which systems it affects. And an "all clear" announcement is posted when the problem has been solved.
If you try and track -stable or -current and do not read <FreeBSD-stable@FreeBSD.ORG> or <FreeBSD-current@FreeBSD.ORG> then you are asking for trouble.
Examine the file /etc/make.conf. This contains some default defines for make, which will be used when you rebuild the source. They are also used every time you use make, so it is a good idea to make sure they are set to something sensible for your system.
Everything is, by default, commented out. Uncomment those entries that look useful. For a typical user (not a developer), you will probably want to uncomment the CFLAGS and NOPROFILE definitions.
Version 2.1.7 and below: If your machine has a floating point unit (386DX, 486DX, Pentium and up class machines) then you can also uncomment the HAVE_FPU line.
This definition was removed for version 2.2.2 and up of FreeBSD.
Examine the other definitions (COPTFLAGS, NOPORTDOCS and so on) and decide if they are relevant to you.
The /etc directory contains a large part of your system's configuration information, as well as scripts that are run at system startup. Some of these scripts change from version to version of FreeBSD.
Some of the configuration files are also used in the day to day running of the system. In particular, /etc/group.
There have been occasions when the installation part of "make world" has expected certain usernames or groups to exist. When performing an upgrade it is likely that these groups did not exist. This caused problems when upgrading.
The most recent example of this is when the "ppp" group (later renamed "network") was added. Users had the installation process fail for them when parts of the ppp subsystem were installed using a non-existent (for them) group name.
The solution is to examine /usr/src/etc/group and compare its list of groups with your own. If they are any groups in the new file that are not in your file then copy them over. Similarly, you should rename any groups in /etc/group which have the same GID but a different name to those in /usr/src/etc/group.
Tip: If you are feeling particularly paranoid, you can check your system to see which files are owned by the group you are renaming or deleting.
# find / -group GID -printwill show all files owned by group GID (which can be either a group name or a numeric group ID).
You may want to compile the system in single user mode. Apart from the obvious benefit of making things go slightly faster, reinstalling the system will touch a lot of important system files, all the standard system binaries, libraries, include files and so on. Changing these on a running system (particularly if you have active users on their at the time) is asking for trouble.
That said, if you are confident, you can omit this step.
Version 2.2.5 and above: As described in more detail below, versions 2.2.5 and above of FreeBSD have separated the building process from the installing process. You can therefore build the new system in multi user mode, and then drop to single user mode to do the installation.
As the superuser, you can execute
# shutdown now
from a running system, which will drop it to single user
mode.
Alternatively, reboot the system, and at the boot prompt, enter the -s flag. The system will then boot single user. At the shell prompt you should then run:
# fsck -p
# mount -u /
# mount -a -t ufs
# swapon -a
This checks the filesystems, remounts / read/write, mounts all the other UFS filesystems referenced in /etc/fstab and then turns swapping on.
As parts of the system are rebuilt they are placed in directories which (by default) go under /usr/obj. The directories shadow those under /usr/src.
You can speed up the "make world" process, and possibly save yourself some dependency headaches by removing this directory as well.
Some files below /usr/obj will have the immutable flag set (see chflags(1) for more information) which must be removed first.
# cd /usr/obj
# chflags -R noschg *
# rm -rf *
You must be in the /usr/src directory, so
# cd /usr/src
(unless, of course, your source code is elsewhere, in
which case change to that directory instead).
To rebuild the world you use the make(1) command. This command reads instructions from the Makefile which describes how the programs that comprise FreeBSD should be rebuilt, the order they should be built in, and so on.
The general format of the command line you will type is as follows;
# make -x -DVARIABLE target
In this example, -x is an option that you would pass to make(1). See the manual page for an example of the options you can pass.
-DVARIABLE passes a variable to the Makefile. The behaviour of the Makefile is controlled by these variables. These are the same variables as are set in /etc/make.conf, and this provides another way of setting them.
# make -DNOPROFILE=true target
is another way of specifying that profiled libaries should not be built, and corresponds with the
NOPROFILE= true
# Avoid compiling profiled libraries
lines in /etc/make.conf.
target tells make(1) what you want to do. Each Makefile defines a number of different "targets", and your choice of target determines what happens.
Some targets are listed in the Makefile, but are not meant for you to run. Instead, they are used by the build process to break out the steps necessary to rebuild the system into a number of sub-steps.
Most of the time you won't need to pass any parameters to make(1), and so your command like will look like this.
# make target
It's a good idea to save the output you get from running make(1) to another file. If something goes wrong you will have a copy of the error message, and a complete list of where the process had got to. While this might not help you in diagnosing what has gone wrong, it can help others if you post your problem to one of the FreeBSD mailing lists.
The easiest way to do this is to use the script(1) command, with a parameter that specifies the name of the file to save all output to. You would do this immediately before remaking the world, and then type exit when the process has finished.
# script /var/tmp/mw.out
Script started, output file is /var/tmp/mw.out
# make world
... compile, compile, compile ...
# exit
Script done, ...
If you do this, do not save the output in /tmp. This directory may be cleared next time you reboot. A better place to store it is in /var/tmp (as in the previous example) or in root's home directory.
/usr/src/Makefile contains the world target, which will rebuild the entire system and then install it.
Use it like this.
# make world
Beginning with version 2.2.5 of FreeBSD (actually, it was first created on the -current branch, and then retrofitted to -stable midway between 2.2.2 and 2.2.5) the world target has been split in two. buildworld and installworld.
As the names imply, buildworld builds a complete new tree under /usr/obj, and installworld installs this tree on the current machine.
This is very useful for 2 reasons. First, it allows you to do the build safe in the knowledge that no components of your running system will be affected. The build is "self hosted". Because of this, you can safely run buildworld on a machine running in multi-user mode with no fear of ill-effects. I still recommend you run the installworld part in single user mode though.
Secondly, it allows you to use NFS mounts to upgrade multiple machines on your network. If you have three machines, A, B and C that you want to upgrade, run make buildworld and make installworld on A. B and C should then NFS mount /usr/src and /usr/obj from A, and you can then run make installworld to install the results of the build on B and C.
The world target still exists, and you can use it exactly as shown for version 2.2.2. make world runs make buildworld followed by make installworld.
Note: If you do the make buildworld and make installworld commands separately, you must pass the same parameters to make(1) each time.
If you run
# make -DNOPROFILE=true buildworldyou must install the results with# make -DNOPROFILE=true installworldotherwise it would try and install profiled libraries that had not been built during the make buildworld phase.
If you are tracking -current you can also pass the -j option to make. This lets make spawn several simultaneous processes.
This is most useful on true multi-CPU machines. However, since much of the compiling process is IO bound rather than CPU bound it is also useful on single CPU machines.
On a typical single-CPU machine you would run:
# make -j4 target
make(1) will then have up to 4 processes running at any one time. Empirical evidence posted to the mailing lists shows this generally gives the best performance benefit.
If you have a multi-CPU machine and you are using an SMP configured kernel try values between 6 and 10 and see how they speed things up.
Be aware that (at the time of writing) this is still experimental, and commits to the source tree may occasionally break this feature. If the world fails to compile using this parameter try again without it before you report any problems.
Assuming everything goes well you have anywhere between an hour and a half and a day or so to wait.
As a general rule of thumb, a 200MHz P6 with more than 32MB of RAM and reasonable SCSI disks will complete make world in about an hour and a half. A 32MB P133 will take 5 or 6 hours. Revise these figures down if your machines are slower...
Remaking the world will not update certain directories (in particular, /etc, /var and /usr) with new or changed configuration files. This is something you have to do by hand, eyeball, and judicious use of diff(1).
You cannot just copy over the files from /usr/src/etc to /etc and have it work. Some of these files must be "installed" first. This is because the /usr/src/etc directory is not a copy of what your /etc directory should look like. In addition, there are files that should be in /etc that are not in /usr/src/etc.
The simplest way to do this is to install the files into a new directory, and then work through them looking for differences.
Backup your existing /etc: Although, in theory, nothing is going to touch this directory automatically, it is always better to be sure. So copy your existing /etc directory somewhere safe. Something like:
# cp -Rp /etc /etc.old-R does a recursive copy, -p preserves times, ownerships on files and suchlike.
You need to build a dummy set of directories to install the new /etc and other files into. I generally choose to put this dummy directory in /var/tmp/root, and there are a number of subdirectories required under this as well.
# mkdir /var/tmp/root
# cd /usr/src/etc
# make DESTDIR=/var/tmp/root distrib-dirs distribution
This will build the necessary directory structure and install the files. A lot of the subdirectories that have been created under /var/tmp/root are empty and should be deleted. The simplest way to do this is to:
# cd /var/tmp/root
# find -d . -type d | /usr/bin/perl -lne \
'opendir(D,$_);@f=readdir(D);rmdir if $#f == 1;closedir(D);'
This does a depth first search, examines each directory, and if the number of files in that directory is 2 ("1" is not a typo in the script) i.e., "." and ".." then it removes the directory.
/var/tmp/root now contains all the files that should be placed in appropriate locations below /. You now have to go through each of these files, determining how they differ with your existing files.
Note that some of the files that will have been installed in /var/tmp/root have a leading ".". At the time of writing the only files like this are shell startup files in /var/tmp/root/ and /var/tmp/root/root/, although there may be others (depending on when you are reading this. Make sure you use ls -a to catch them.
The simplest way to do this is to use diff(1) to compare the two files.
# diff /etc/shells /var/tmp/root/etc/shells
This will show you the differences between your /etc/shells file and the new /etc/shells file. Use these to decide whether to merge in changes that you have made or whether to copy over your old file.
Name the new root directory (/var/tmp/root)with a timestamp, so you can easily compare differences between versions: Frequently remaking the world means that you have to update /etc frequently as well, which can be a bit of a chore.
You can speed this process up by keeping a copy of the last set of changed files that you merged into /etc. The following procedure gives one idea of how to do this.
Make the world as normal. When you want to update /etc and the other directories, give the target directory a name based on the current date. If you were doing this on the 14th of February 1998 you could do the following.
# mkdir /var/tmp/root-19980214 # cd /usr/src/etc # make DESTDIR=/var/tmp/root-19980214 \ distrib-dirs distributionMerge in the changes from this directory as outlined above.
Do not remove the /var/tmp/root-19980214 directory when you have finished.
When you have downloaded the latest version of the source and remade it, follow step 1. This will give you a new directory, which might be called /var/tmp/root-19980221 (if you wait a week between doing updates).
You can now see the differences that have been made in the intervening week using diff(1) to create a recursive diff between the two directories.
# cd /var/tmp # diff -r root-19980214 root-19980221Typically, this will be a much smaller set of differences than those between /var/tmp/root-19980221/etc and /etc. Because the set of differences is smaller, it is easier to migrate those changes across into your /etc directory.
You can now remove the older of the two /var/tmp/root-* directories.
# rm -rf /var/tmp/root-19980214Repeat this process every time you need to merge in changes to /etc.
You can use date(1) to automate the generation of the directory names.
# mkdir /var/tmp/root-`date "+%Y%m%d"`
DEVFS: If you are using DEVFS then this is probably unnecessary.
For safety's sake, this is a multistep process.
Copy /var/tmp/root/dev/MAKEDEV to /dev.
# cp /var/tmp/root/dev/MAKEDEV /dev
Now, take a snapshot of your current /dev. This snapshot needs to contain the permissions, ownerships, major and minor numbers of each filename, but it should not contain the timestamps. The easiest way to do this is to use awk(1) to strip out some of the information.
# cd /dev
# ls -l | awk '{print $1, $2, $3, $4, $5, $6, $NF}' > /var/tmp/dev.out
Remake all the devices.
# sh MAKEDEV all
Write another snapshot of the directory, this time to /var/tmp/dev2.out. Now look through these two files for any devices that you missed creating. There should not be any, but it is better to be safe than sorry.
# diff /var/tmp/dev.out /var/tmp/dev2.out
You are most likely to notice disk slice discrepancies which will involve commands such as
# sh MAKEDEV sd0s1
to recreate the slice entries. Your precise
circumstances may vary.
Note: This step is included only for completeness, it can safely be omitted.
For completenesses sake you may want to update the files in /stand as well. These files consist of hard links to the /stand/sysinstall binary. This binary should be statically linked, so that it can work when no other filesystems (and in particular /usr) have been mounted.
# cd /usr/src/release/sysinstall
# make all install
Source older than 2 April 1998: If your source code is older than 2nd April 1998, or the Makefile version is not 1.68 or higher (for FreeBSD current and 3.x systems) or 1.48.2.21 or higher (for 2.2.x systems) you will need to add the NOSHARED=yes option, like so;
# make NOSHARED=yes all install
To take full advantage of your new system you should recompile the kernel. This is practically a necessity, as certain memory structures may have changed, and programs like ps(1) and top(1) will fail to work until the kernel and source code versions are the same.
Follow the handbook instructions for compiling a new kernel. If you have previously built a custom kernel then carefully examine the LINT config file to see if there are any new options which you should take advantage of.
A previous version of this document suggested rebooting before rebuilding the kernel. This is wrong because:
Commands like ps(1), ifconfig(8), and sysctl(8) may fail. This could leave your machine unable to connect to the network.
Basic utilities like mount(8) could fail, making it impossible to mount /, /usr and so on. This is unlikely if you are tracking a -stable candidate, but more likely if you are tracking -current during a large merge.
Loadable kernel modules (LKMs on pre-3.x systems, KLDs on 3.x systems and above) built as part of the "world" may crash an older kernel.
For these reasons, it is always best to rebuild and install a new kernel before rebooting.
You should build your new kernel after you have completed make world (or make installworld). If you do not want to do this (perhaps you want to confirm that the kernel builds before updating your system) you may have problems. These may be because your config(8) command is out of date with respect to your kernel sources.
In this case you could build your kernel with the new version of config(8)
# /usr/obj/usr/src/usr.sbin/config/config KERNELNAME
This may not work in all cases. It is recommended that you complete make world (or make installworld) before compiling a new kernel.
You are now done. After you have verified that everything appears to be in the right place you can reboot the system. A simple fastboot(8) should do it.
# fastboot
You should now have successfully upgraded your FreeBSD system. Congratulations.
You may notice small problems due to things that you have missed. For example, I once deleted /etc/magic as part of the upgrade and merge to /etc, and the file command stopped working. A moment's thought meant that
# cd /usr/src/usr.bin/file
# make all install
was sufficient to fix that one.
A: There is no easy answer to this one, as it depends on the nature of the change. For example, I have just run CVSup, and it has shown the following files as being updated since I last ran it;
src/games/cribbage/instr.c
src/games/sail/pl_main.c
src/release/sysinstall/config.c
src/release/sysinstall/media.c
src/share/mk/bsd.port.mk
There is nothing in there that I would re-make the world for. I would go to the appropriate sub-directories and make all install, and that's about it. But if something major changed, for example src/lib/libc/stdlib then I would either re-make the world, or at least those parts of it that are statically linked (as well as anything else I might have added that is statically linked).
At the end of the day, it is your call. You might be happy re-making the world every fortnight say, and let changes accumulate over that fortnight. Or you might want to re-make just those things that have changed, and are confident you can spot all the dependencies.
And, of course, this all depends on how often you want to upgrade, and whether you are tracking -stable or -current.
A: This is normally indicative of hardware problems. (Re)making the world is an effective way to stress test your hardware, and will frequently throw up memory problems. These normally manifest themselves as the compiler mysteriously dying on receipt of strange signals.
A sure indicator of this is if you can restart the make and it dies at a different point in the process.
In this instance there is little you can do except start swapping around the components in your machine to determine which one is failing.
A: That depends on how you want to make the world on future occasions.
/usr/obj contains all the object files that were produced during the compilation phase. Normally, one of the first steps in the "make world" process is to remove this directory and start afresh. In this case, keeping /usr/obj around after you have finished makes little sense, and will free up a large chunk of disk space (currently about 150MB).
However, if you know what you are doing you can have "make world" skip this step. This will make subsequent builds run much faster, since most of sources will not need to be recompiled. The flip side of this is that subtle dependency problems can creep in, causing your build to fail in odd ways. This frequently generates noise on the FreeBSD mailing lists, when one person complains that their build has failed, not realising that it is because they have tried to cut corners.
If you want to live dangerously then make the world, passing the NOCLEAN definition to make, like this:
# make -DNOCLEAN world
A: This depends on how far through the process you got before you found a problem.
In general (and this is not a hard and fast rule) the "make world" process builds new copies of essential tools (such as gcc(1), and make(1)>) and the system libraries. These tools and libraries are then installed. The new tools and libraries are then used to rebuild themselves, and are installed again. The entire system (now including regular user programs, such as ls(1) or grep(1)) is then rebuilt with the new system files.
If you are at the last state, and you know it (because you have looked through the output that you were storing) then you can (fairly safely) do
... fix the problem ...
# cd /usr/src
# make -DNOCLEAN all
This will not undo the work of the previous "make world".
If you see the message
--------------------------------------------------------------
Building everything..
--------------------------------------------------------------
in the "make world" output then it is probably fairly
safe to do so.
If you do not see that message, or you are not sure, then it is always better to be safe than sorry, and restart the build from scratch.
A: People often ask on the FreeBSD mailing lists whether they can do all the compiling on one machine, and then use the results of that compile to make install on to other machines around the network.
This is not something I have done, so the suggestions below are either from other people, or deduced from the Makefiles.
The precise approach to take depends on your version of FreeBSD
You must still upgrade /etc and /dev on the target machines after doing this.
For 2.1.7 and below, Antonio Bemfica suggested the following approach:
Date: Thu, 20 Feb 1997 14:05:01 -0400 (AST)
From: Antonio Bemfica <bemfica@militzer.me.tuns.ca>
To: freebsd-questions@freebsd.org
Message-ID: <Pine.BSI.3.94.970220135725.245C-100000@militzer.me.tuns.ca>
Josef Karthauser asked:
> Has anybody got a good method for upgrading machines on a network
First make world, etc. on your main machine
Second, mount / and /usr from the remote machine:
main_machine% mount remote_machine:/ /mnt
main_machine% mount remote_machine:/usr /mnt/usr
Third, do a 'make install' with /mnt as the destination:
main_machine% make install DESTDIR=/mnt
Repeat for every other remote machine on your network. It works fine
for me.
Antonio
This mechanism will only work (to the best of my knowledge) if you can write to /usr/src on the NFS server, as the install target in 2.1.7 and below needed to do this.
Midway between 2.1.7 and 2.2.0 the "reinstall" target was committed. You can use the approach exactly as outlined above for 2.1.7, but use "reinstall" instead of "install".
This approach does not require write access to the /usr/src directory on the NFS server.
There was a bug introduced in this target between versions 1.68 and 1.107 of the Makefile, which meant that write access to the NFS server was required. This bug was fixed before version 2.2.0 of FreeBSD was released, but may be an issue of you have an old server still running -stable from this era.
For version 2.2.5 and above, you can use the "buildworld" and "installworld" targets. Use them to build a source tree on one machine, and then NFS mount /usr/src and /usr/obj on the remote machine and install it there.
Q: How can I speed up making the world?
Run in single user mode.
Put the /usr/src and /usr/obj directories on separate filesystems held on separate disks. If possible, put these disks on separate disk controllers.
Better still, put these filesystems across separate disks using the "ccd" (concatenated disk driver) device.
Turn off profiling (set "NOPROFILE=true" in /etc/make.conf). You almost certainly do not need it.
Also in /etc/make.conf, set "CFLAGS" to something like "-O -pipe". The optimisation "-O2" is much slower, and the optimisation difference between "-O" and "-O2" is normally negligible. "-pipe" lets the compiler use pipes rather than temporary files for communication, which saves disk access (at the expense of memory).
Pass the -j<n> option to make (if you are running a sufficiently recent version of FreeBSD) to run multiple processes in parallel. This helps regardless of whether you have a single or a multi processor machine.
The filesystem holding /usr/src can be mounted (or remounted) with the "noatime" option. This stops the time files in the filesystem were last accessed from being written to the disk. You probably do not need this information anyway.
Note: "noatime" is in version 2.2.0 and above.
# mount -u -o noatime /usr/src
Warning: The example assumes /usr/src is on its own filesystem. If it is not (if it is a part of /usr for example) then you will need to use that filesystem mount point, and not /usr/src.
The filesystem holding /usr/obj can be mounted (or remounted) with the "async" option. This causes disk writes to happen asynchronously. In other words, the write completes immediately, and the data is written to the disk a few seconds later. This allows writes to be clustered together, and can be a dramatic performance boost.
Warning: Keep in mind that this option makes your filesystem more fragile. With this option there is an increased chance that, should power fail, the filesystem will be in an unrecoverable state when the machine restarts.
If /usr/obj is the only thing on this filesystem then it is not a problem. If you have other, valuable data on the same filesystem then ensure your backups are fresh before you enable this option.
# mount -u -o async /usr/obj
Warning: As above, if /usr/obj is not on its own filesystem, replace it in the example with the name of the appropriate mount point.
The following people have contributed to this document in some form or another. Either by directly suggesting alterations and improvements, pointing out errors, or by their messages to the FreeBSD mailing lists, from which I have shamelessly cribbed information. My thanks to them.
Antonio Bemfica, <bemfica@militzer.me.tuns.ca>
Sue Blake, <sue@welearn.com.au>
Brian Haskin, <haskin@ptway.com>
Kees Jan Koster, <kjk1@ukc.ac.uk>
A Joseph Kosy, <koshy@india.hp.com>
Greg Lehey, <grog@lemis.com>
Wes Peters, <softweyr@xmission.com>
Joseph Stein, <joes@wstein.com>
Studded, <studded@dal.net>
Axel Thimm, <Axel.Thimm@physik.fu-berlin.de>
Matthew Thyer, <Matthew.Thyer@dsto.defence.gov.au>