nmtools Guide - by t00fri

A tutorial by Fridger on how to use this software.

    Table of contents:
  1. Introduction and Acknowledgements
  2. t;>Purpose of Normal Maps in a 'Nutshell'
  3. Generating Huge Normal Maps Yourself!
    1. 3.1. The input height map
    2. 3.2. The nmtools functionalities
    3. 3.3. Preparing the console in Windows XP
    4. 3.4. Installing the Tools
    5. 3.5. A small-size testrun
    6. 3.6. Step-by-step recipe for the 'real thing'
  4. An appropriate 64k BMNG base texture

 



1: Introduction and Acknowledgements

In this article I am attempting a concise tutorial about the use of our new sophisticated software tools that I called 'nmtools'. They serve to produce quickly highest quality normal maps of HUGE size for the 3d-space simulation Celestia on average home computers!

Some time ago, Robert Skuridin and I decided to join our know-how for this interesting and challenging task. Like me, Robert is a theoretical physicist. He is from Russia and works in hydrodynamics. He has special expertise in precision algorithms, however, using mainly FORTRAN for Windows as his coding language. Notably, Robert was the first to notice that all previous normal map code was badly incorrect for (approximate) spherical geometry, as used of course for planets etc in Celestia!

For the C++ coding, the cross-platform implementation and for all the rest in the nmtools archive I am to blame...

As the result of our intensive collaboration about optimizing the algorithms, the tools are distinguished by the following desirable features:

  • they produce e.g.a complete set of highest quality normal map tiles, in less than ~20 minutes on current computers, directly from the huge published, professional elevation maps in (signed) 16bit = 2Byte raw format. Both byte-orderings (so-called big/little-endian ordering of the 2 Bytes) are supported. All is done IN ONE "custom" STEP.
  • All known optimizations and constraints are incorporated, like the proper resolution reduction for polar latitudes, the replacement of monochromatic tiles (oceans!) by the smallest possible 4x4 pixel tiles and correct texel normalization (r*r + g*g + b*b = 1) throughout! The optimizations help crucially to reduce the storage requirements for these big textures and partly increase the stability of operation...
  • For the first-time - the crucial spherical geometry correction (planets,...) is taken into account. It's effect is increasingly crucial towards the north and south poles!
  • Binary executables for all popular platforms: Windows XP, Mac(PPC), Mac(Intel) and Linux are included in the distribution, along with the source code according to the GPL licence.
  • The tools are most carefully tested to work robustly on average low memory home computers, even in case of huge normal maps (of >= 65536 x 32768 pixels!)
  • No need for special computer or image manipulation expertise or for installing external libraries!

A number of friends from our Celestia developer team helped with testing and gave most useful feedback:

I cordially thank Da Woon Jung (aka Dirkpitt) for his active help to make the code run on Macs. He compiled the tools as "OSX-universal", meaning they contain both PPC and Intel code and run natively on PPC and Intel Macs.

Many thanks also go to Chris Laurel for a code sniplet allowing redirection of STDIN and STDOUT for Windows in case of binary file input/output. I am also pleased to acknowledge his detailed feedback and further ideas for the future.

Last not least, I thank Christophe Teyssier for his extensive pro-style benchmarking of the tools, along with a host of concrete tuning proposals for the next update that will make the tools even faster!



2: Purpose of Normal Maps in a 'Nutshell'

A detailed introduction into normal maps would certainly exceed the scope of this tutorial. Here are nevertheless some basic, respective comments. "Normal maps in a nutshell", so to speak...

For those of you who now ask what normal maps are good for, let me refer to this nice tutorial: http://planetpixelemporium.com/tutorialpages/normal.html, and also to this one: http://www.bencloward.com/tutorials_normal_maps1.shtml.

While a height map only associates simple numbers (the heights) with each point of a given surface, each pixel of a normal map (color-)encodes which direction in space that particular surface point is facing. This requires specification of the components of a 3-dimensional (unit) vector - the "normal vector" at each point (x,y) of the surface. It is orthogonal to the surface as it's name says.

The subject of this article concerns the fact that normal maps may be calculated from given height maps by encoding the change in height in both x and y directions in a standardized manner. That rather complex task is done quickly and simply for any operating system with the help of our new software tools.

In Celestia, we overlay the normal maps over the base texture that contains essentially the color information of the celestial object's surface under consideration (planet, asteroid, etc). The purpose of the normal maps is to create a sense of dynamical shading of the surface, i.e. a variable shading, depending on the position of the light source (Sun, other star).

The vector product of the normal vector at each surface point with the "light vector" characterizing the direction to the light source then specifies the brightness of that point. This is a fancy way of saying that the illumination of a surface point is high if the angle between the light direction and normal vector direction is small. The illumination decreases if that angle increases.Thus, dynamical shading is at hand given the normal map texture!

In various respects, normal maps represent the most important textures for the composite hires rendering of planetary surfaces. Let's take a moment to convince ourselves about the importance of normal maps:

normalmap normalmap normalmap
Fig. 1 Fig. 1a Fig. 1b
(click on any picture to open it full-size in a new window)

Fig. 1 illustrates what Celestia would display, if we had discarded ALL textures BUT the normal map in the file solarsys.ssc! In fact, that picture is a tiny window to the huge normal map with 65536 x 32768 pixels (64k) that you are about to generate yourself in this tutorial!

In fig. 1, we observe a tremendous amount of detail, with the flat Pacific in the lower left corner and the rugged and most detailed mountain landscape of the South American Cordilleres in the remaining parts of the picture.

Next, figs. 1a and 1b impressively demonstrate how the base texture and the normal map cooperate to form a final, most detailed image:

The gray, top image in fig. 1a is what Celestia shows of some mountaineous region in central Asia, using just again our 64k normal map. The corresponding view with ONLY the base texture (color) activated, is displayed in the bottom image of fig. 1a. Despite the equally high resolution (64k), it looks much less impressive by itself. Fig.1b then displays the highly detailed result of overlaying the color texture with the normal map. It is of crucial importance that the base texture and the normal map match precisely!

A most crucial requirement for hires normal maps is their high quality, meaning smoothness and the lack of noise! Apart from ongoing promising experiments related to modified DXT5 encoding, the familiar, lossless PNG format is still the preferable high quality choice, notably for normal-map tiles, given a sufficiently powerful computer.

In order to reduce the noise to an acceptable level, the normal-map calculations have to be done at full 16bit level of the input elevation maps.

Remember that a 16bit = 2Byte integer allows to encode 65536 = 2^16 different height values, while a 8bit = 1Byte number can only represent 2^8 = 256 height values. For highest accuracy (i.e. smoothness) and to minimize noise, it is crucial that truncation to the 8bit rgb normal map format is only performed at the very end. This crucial idea was first realized in Chris Laurel's normal map program, nm16, which however also fails to account for the spherical geometry correction. In our nmtools kit, all operations before calculating the normal map are done at the 16bit level in order to maximize the resulting quality, to minimize noise and to retain proper normalization of the calculated normal vector.

 



3: Generating Huge Normal Maps Yourself!

A detailed introduction into normal maps would certainly exceed the scope of this tutorial. Here are nevertheless some basic, respective comments. "Normal maps in a nutshell", so to speak...

After these prerequisites, we are now ready to turn to the main task, the generation of huge normal maps by means of our new nmtools. For this tutorial, I shall concentrate on making the latest and best "monster" normal map of 65536 x 32768 pixels (64k) for our planet in form of 1k x 1k optimized tiles.

However, before starting, I have to make some important comments!

These grew out of some recent frustrating experiences I made in the Celestia Forum, where I tried to present some respective warming-up "exercises", notably of how to use the Windows console along with some simple command-line commands.

Originally, I planned to present the nmtools tutorial at a level that assumes users to be familiar with the essential features and functionalities of their own operating system. Obviously, the purpose of this tutorial cannot be to teach people how to use their computers, since this would vastly exceed the scope of this article! Notably since there exist many excellent books about Windows XP for example.

fig2
Fig. 2

In addition, I had planned to teach people about some elegant, less known command-line features that exist equally in all 4 supported operating systems: the so-called STDIN/STDOUT redirection along with the possibility to plug several command-line tools together like pipe fittings as illustrated in [fig2] in a self-explanatory manner. The nmtools all make use of these great functionalities.

It turned out, however, that many interested users in the Forum were even unaware, how to locate the standard console application in Windows XP, for example; or how to place it's corresponding icon on the desktop for future use.

Yet, the essential problematics was that most did NOT follow precisely the instructions that I had explicitly written down in a step-by-step manner. It was also obvious that my instructions were at best read in a quick, superficial manner.

At the end, everything was of course considered to be my fault, in the sense of being not sufficiently pedagogical...

In view of this background problematics, I have decided to change my plans as follows:

  • Unlike my original intention, I shall skip most attempts to provide an understanding of the used functionalities, since I learned that many are not really interested to know.
  • As a consequence, I shall largely present a step-by-step sequence of instructions (Sect. 3.6) that will lead to success if followed precisely. If not, users will be on their own.
  • Since reading of somewhat longer articles is often considered a pain, the explanations in Sect. 3.6 below were deliberately shrunk to a minimum.

People with absolute disinterest in the background information that I shall provide while going along, can therefore directly jump to Sect. 3.6.

3.1: The input height map
So you know what you will be downloading, let me nevertheless spend some comments on the required input file, the published high-quality 'srtm_ramp2.world' elevation map of our planet.

It contains in total 86400 x 43200 = 373 248 000 (!) height values, stored for each surface point (in simple cylindrical projection) as 16 bit = 2 Byte (signed) integers. The units are meters and the range is accordingly from -32768 m to +32768 m, which is certainly sufficient for characterizing smoothly all possible elevations on Earth. The sea level is put at 0 m. Unpacked, this map is a 8 GB monster! But in the compressed form that you will download, it's "only" 1.2 GB long...

Note that height maps of other planets, like e.g. Mars, are published in the same format and, correspondingly, can be processed equally well with our nmtools!

The only difference is possibly the byte ordering, in which the 2 Byte heights are stored in the input elevation map. One distinguishes so-called "big endian" and "little endian" byte ordering as follows:

  • Little endian means that the low-order byte of the number is stored in memory or on file at the lowest address, and the high-order byte at the highest address. (The little end comes first.) For example, a 2 Byte integer (Byte1 Byte0) will be arranged as in fig. 3 (top). Intel processors (those used in PC's and Intel Macs) use "Little Endian" byte order.
  • Big endian means that the high-order byte of the number is stored in memory or on file at the lowest address, and the low-order byte at the highest address. (The big end comes first.) Our 2 Byte Int, would then be stored as in fig. 3b (bottom). Motorola processors (those used in PPC Mac's) use "Big Endian" byte order.
fig3
Fig. 3

The great point is that the elevation map data are precisely those that Reto Stoeckli (NASA) and his collaborators used to (statically) shade the BMNG base texture! Remember, BMNG = Blue Marble Next Generation, the best available base texture of Earth. In other words our elevation map data will match the BMNG base texture perfectly!

 

  • 3 arc-second = 90 m resolution SRTM dataset (Shuttle Radar Topography Mission, [JPL, 2005]) from 60S-60N degrees. The SRTM instrument consisted of the Spaceborne Imaging Radar-C (SIR-C) hardware set modified with a Space Station-derived mast and additional antennae to form an interferometer with a 60 meter long baseline.
  • 30 arc-second GTOPO30 dataset [USGS, 1996], from 60N-90N degrees, and to fill small voids in the SRTM dataset, using bi-cubic interpolation.
  • RAMP II dataset (Radarsat Antarctic Mapping Project Digital Elevation Model Version 2, [Liu et al., 2001]) from 90S-60S degrees.

3.2: The nmtools functionalities
The nmtools distribution presently consists of 4 distinct command-line tools whose basic functionalities I want to briefly introduce in this sub-section. I underwent special efforts to achieve that in all 4 supported operating systems (Win XP, Mac(PPC), Mac (Intel) and Linux), the SAME command syntax is to be used!

Firstly, all 4 nmtools utilities support some elegant, less known command-line features that exist equally in all 4 supported operating systems: The so-called 'STDIN/STDOUT redirection'. This means that the input/output to/from these 4 utilities proceeds via standard input/output channels, called generically STDIN/STDOUT.

On the one hand, this elegant feature allows to plug several of our utilities together like pipe fittings, by means of the pipe operator ' | '. This is symbolically illustrated in a self-explanatory manner in fig. 2. Note that no intermediate output file storage is then necessary!

On the other hand, the STDIN/STDOUT channels may be 'redirected' to any desired input/output filenames with the redirection operators ' < ' (input) and ' > ' (output).

Generic example:

Suppose we have 3 utilities called program1, program2 and program3. Moreover let 'infile' be the name of our input file for program1, and 'outfile' the filename we like to use for storing the output data from program3. Then using the ' | ' pipe and ' < ', ' > ' redirection operators, we may write in all 4 supported operating systems equally at the console prompt (cf. also fig2!):

program1 < infile | program2 | program3 > outfile

It should now be intuitively obvious what this command line will be doing for you:

program1 reads the input file 'infile', does some calculations with it and produces some output into its STDOUT channel. This in turn is piped via ' | ' to STDIN of program2 which continues to do calculations, now with the output from program1 etc. Finally, the desired output from program3 will be stored in 'outfile'.

This command line is still a bit oversimplified, since in general, each of the utilities rquires some additional parameters to work correctly as desired. You will see the finalized form of these commands further down.

Here is a brief summary of the purpose and functionalities of our 4 tools:

  1. resc2pow2

    The program reads samples with signed 16 bit integers from STDIN and outputs to STDOUT a signed 16 bit integer sample of width reduced to the nearest power of 2.

    Specify byteswap = 1 on the command line if the byte ordering of input file and computer differ.

    Example: The utility 'resc2pow2' reduces a height map of size 86400 x 43200 to one of size 65536 x 32768 = 2^16 x 2^15 (i.e. a 64k x 32k texture).
  2. halfsize

    The program reads textures of even height in signed 16 bit integer raw format from STDIN. It outputs to STDOUT a texture of size reduced by a factor of two in the same 16 bit raw format.

    Specify byteswap = 1 if the byte ordering of input file and computer differ.

    Example: if you want to reduce the power-of-two size of the file 'infile' by a factor of 8, say, you may 'pipe' 3 'halfsize' calls together like so (ignoring again further required parameters):

    halfsize < infile | halfsize | halfsize > outfile

    The file 'outfile' then stores the new 16 bit texture rescaled by a factor of 8 in width.
  3. nms

    The program reads an elevation map in signed 16 bit raw format from STDIN. It outputs to STDOUT a normalmap (PPM format), for bodies of ~spherical geometry. All previously published normalmap programs (including Chris Laurel's nm16) are incorrect for ~spherical bodies north and south of the equator.
  4. nmstiles

    This is the most important and most sophisticated utility of the distribution.


    The program reads an elevation map in signed 16 bit raw format from STDIN. It directly outputs normalmap tiles (PPM format), corrected for spherical geometry and with many optimizations. These optimizations include: the appropriate reduction of resolution for polar latitudes, correct texel normalization (r*r + g*g + b*b = 1) and the replacement of monochromatic normalmap tiles by the smallest possible 4 x 4 pixel tiles! The latter saves a large amount of disc space without any loss of quality.

    It is assumed that the input elevation file is of power-of-two size

    Example: If the published elevation file is not of power-of-two size you may 'pipe' 'nmstiles' together with 'resc2pow2' as follows:

    resc2pow2 < infile | nmstiles > outfile.ppm

3.3: Preparing the console in Windows XP
Next, let me account briefly of the fact that many Windows XP users seem to be little aquainted with the standard command-line console ('cmd.exe') (Why?? Probably because Bill Gates told you that good Windows users should never do anything else but 'click'...? ). The other operating systems (Mac OSX and Linux) also offer a console with a powerful command shell ('bash'), along with a UNIX directory structure and standard UNIX commands. I simply assume that Mac and Linux users KNOW how to activate this important tool.

In Windows, a simple way of activating the console, is by clicking Start-> Run..., typing 'cmd' into the run dialog and hitting OK. The console window appears, by default with a white font on a black background.

Since the console will be used frequently, I propose to place a console icon on the desktop, allowing to open the console by clicking on it. Also, you may customize this way the appearance of the console window. All these tasks are done as follows:

  • Conveniently, with your Windows Explorer, change to the Windows system directory, c:\WINDOWS\system32
  • After locating 'cmd.exe', click it with the right-hand mouse key
  • In the resulting popup, click Send To Desktop (create shortcut)

Then you may rename the resulting Desktop icon e.g. to 'Console' by clicking it again with the right-hand mouse key and then selecting 'Rename'. If you finally want to customize your Console a bit, select 'Properties' after clicking it once more with the right-hand mouse key. I prefer the 'Lucida Console' font. You'll have to adjust the font size to your gusto, depending on your monitor resolution. I also prefer a blue background instead of the black default one. This you may set in the 'Colour' section. In the 'Layout' section you may adjust the window size of the console. I usually set 35 lines instead of the default height of 25 lines.

People who don't know the standard DOS commands, may type 'help' at the console prompt which results in a listing of all recognized commands. If you want to have more info about a particular command, type 'help <command>'.

Example:
A change of directories is done with the DOS command 'chdir' or shorter 'cd'. By typing 'help cd' you can remind yourself of the precise respective command syntax. Note: DOS console commands are NOT case sensitive, unlike Windows or Linux commands.

3.4: Installing the tools
After downloading and unpacking the nmtools archive, please stick to the following installation instructions precisely. Notably for Windows users, I shall assume that you have downloaded an nmtools version >= 1.0.1

Windows:

  • In the nmtools directory Win32_PC.bin, click Setup.exe and follow the instructions.
  • Unlike version 1.0, in version 1.0.1 and later the Inno Setup script also places appropriate entries in the Windows Registry that make the nmtools known as executable programs in any directory, just by typing the name of the desired tool at the console prompt.
  • In order to activate the new Registry entry, the simplest option is to reboot your computer after installing the nmtools!!!
  • Alternatively, to avoid rebooting, you may open the environment dialog via Start->Control Panel->System->Advanced->Environment Variables
  • Then control whether in the top scrolled listing of 'User variables for <user>' the Pathvariable exists and contains the entry 'C:\Program Files\Nmtools' (once or several times separated by semicolons). If so, hit OK OK. This should also activate the Registry entry without rebooting.

After these preparations, let us perform some crucial tests of your nmtools installation!

  • Open the console by clicking on the respective Desktop icon.
  • Type at the command prompt the name of one of the nmtools, like e.g. 'nms'.
  • You should see a minimalistic help text being printed along with version number and the authors.

If you don't get that help printout, something is NOT working and you must find the reason before proceeding!

Linux and Mac OSX

  • Please open your console, become 'root' and change to the nmtools package subdirectory 'src' from there. Then type at the prompt:

    - LINUX: 'make install'
    - Mac OSX: 'make -f Makefile.osx install'

These commands will install the 4 nmtools in the directory /usr/local/bin, which represents the standard execution path for local executables in any UNIX directory structure. As a check, become a normal user again and type e.g. 'nms' at the console prompt in your home directory. You should see a minimalistic help text about how to use 'nms', along with the version and the authors.

If you don't see that help text, something went wrong and you must find out the reason before proceeding!

3.5: A small-size testrun
Now we are all set and will perform a first realistic warm-up test with the nmtools! Notably, in this section, I shall also explain all the steps of converting our normalmap output in PPM format to various desirable formats that are directly supported by Celestia! This implies some simple installing steps of well-known (freeware) texture format converters.

In order to get started, please download the small 4k elevation file below that I have prepared for you by means of the nmtools. It has the same 16 bit integer raw format as the huge 64k 'srtm_ramp2.world.86400 x 43200.bin' heightmap.

There is one basic difference, though: I saved it in 'little endian' storage mode unlike its 64k predecessor, which is stored in 'big endian' mode (cf. sect. 3.1). As I wrote above, this means 'little endian' PC's (Linux, Win32, Intel Macs) do NOT have to add the option 'byteswap = 1' in the command line of our test example, while 'big endian' Macs do have to include a byte-swap option.

Here is the 4k test heightmap for download: srtm_ramp2.world.4096x2048.bin.zip [3,9 MB].

  • Download it by clicking on the link and unzip that binary heightmap file into some convenient directory ( e.g. in Win32: 'My Documents' or in Linux, Mac OSX: your home directory).
  • Open the console and change to the directory where you unzipped the above file.

The first exercise is very simple:
We shall produce a nice (exaggerated) normal map from our 4k input heightmap by using the tool 'nms'. Try understanding the command by comparing with the general syntax printed out by typing 'nms' without arguments.

nms 6378.140 4096 20.0 < srtm_ramp2.world.4096x2048.bin > nm4k.ppm

Note: People with a 'big endian' Mac MUST add the byte-swap option, like so:

nms 6378.140 4096 20.0 1 < srtm_ramp2.world.4096x2048.bin > nm4k.ppm

Examine the resulting nm4k.ppm file with your favorite image display program. On my machine this normalmap generation takes very little time, about 1.5 sec ONLY... Not bad for a 4k texture!

Note that I have used exag = 20.0 which corresponds to a large height exaggeration in order to see the normal map more clearly at such low resolutions... For the really large normalmaps you should rather use exag = 2.5 or similar.

Using the elegant Pipe mechanism:
After this most elementary application of the tools, let us next explore the elegant pipe mechanism, described above. Try understanding what happens here, before reading my explanations below!

halfsize 4096 < srtm_ramp2.world.4096x2048.bin | nms 6378.140 2048 20.0 > nm2k.ppm

Note: People with a 'big endian' Mac MUST add the byte-swap option like so

halfsize 4096 1 < srtm_ramp2.world.4096x2048.bin | nms 6378.140 2048 20.0 > nm2k.ppm

If at all necessary, the byte-swap option is only required in the first tool in the pipe. Outputs (on STDOUT) are always done in the machine's 'natural' byte ordering, hence further byte-swaps are unnecessary. Remember, the byte-swap parameter is required whenever the storage mode of the input heightmap file differs from the endedness of the respective computer! In our present test case this only happens for 'big endian' Macs. The new Intel Macs, however, are 'little endian' machines and thus would NOT need a byte-swapping entry in case of our 'little endian' input file. Got it?

For the actual 85k heightmap (cf. next section) everything is reversed as to byte-swapping, since the storage mode of the original 85k heightmap is 'big endian' unlike our little testfile!

As explained generically above, the tool 'halfsize' reduces the input heightmap by a factor of 2 at the 16 bit level and feeds its output into 'nms' via the pipe operator ' | '. The final result is then stored into the normalmap file nm2k.ppm in lossless PPM format (='Portable PixMap'). Hence this generated normalmap is smaller by a factor of 2 compared to our first one.

The lossless standard 'portable pixmap' output PPM format may be easily converted into whatever you like by any image manipulation program including nconvert (XnView), GIMP, Photoshop,.. . Hence, I did not vaste my time on coding output modules for other image formats. For the small-sized VT tiles, the best quality normalmap format is still the lossless PNG format.

Let me add next some general advice about how to convert our .ppm normalmaps to a format supported by Celestia in the various operating systems:

For all operating systems, I strongly recommend the command-line tool 'nconvert' that is part of the well-knownfreeware multi-language, multi-OS XnView distribution. Installation is trivial, and only takes a few minutes including download. nconvert supports a HUGE number of format conversions and thus is a most useful tool to have on your machine.

Windows:
For using 'nconvert' conveniently from the console, you must again make it known as an executable in any application directory. Since the XnView installer does not do a corresponding Registry entry, we must make 'nconvert' known ourselves as follows:

  • Open the 'Environment variables' dialog via Start -> Control Panel -> System -> Advanced -> Environment Variables
  • In the listing of 'System variables' select the variable 'Path' and click Edit. Click into the line named 'Variable value' and add a semi-colon behind the last entry. Leave everything as it is for now!
  • Next, with your Windows Explorer, change to the XnView installation directory C:\Program Files\XnView
  • Copy this directory name with CTRL+C in the WE directory line and insert it with CTRL+V behind the semi-colon in the 'variable value' line of the 'Edit system variables' dialog. If you prefer, you may as well directly type in the directory line. Yet my method via WE tends to avoid typos...
  • Hit OK OK OK.
  • Open the console in an arbitrary directory and type 'nconvert' at the prompt. As with the nmtools, you now must see a mini help output! Otherwise, something went wrong.

Please follow my instruction carefully. This is a basic way of making command-line programs generally known in application directories. You may and actually SHOULD apply the same procedure for any other command-line program (that lacks a corresponding Registry entry via its installer)!

Next some advice about conversion to the new compressed DXT5NM format that is supported since Celestia (pre x) 1.5.0.

For Windows users, NVIDIA's 'nvDXT' command-line tools are best for any conversion jobs into variants of the DXT format. While they only work with small textures of size <= 8k x 4k, they are perfect for VT tiles! The NVIDIA DDS utilities come with a simple installer. It also takes care of making the tools known in all application directories after a reboot! Check it by opening the console and typing 'nvdxt' into an arbitrary directory!

Here are the 2 console commands that you may use to convert whole levelx, x=0,1,... directories of *.ppm normalmap tiles to the new *.dxt5nm format, for example. Just chdir to each levelx directory with the console and call this command:

  • nvdxt -quality_highest -file *.ppm -nomipmap -dxt5nm

    After nvdxt has finished in each sub-directory, rename all *.dds file output into *.dxt5nm, as required by Celestia 1.5.0. Type at the console prompt:

    ren *.dds *.dxt5nm
  • </ul>Note this new dxt5nm format is only recognized by Celestia >= (pre x) 1.5.0 and needs to be assigned to in the respective .ctx file as usual. It has many advantages, though, without too many compromises as to image quality.

    Linux, Mac-OSX
    After installation, Linux and Mac OSX users should be able to call 'nconvert' from any application directory just by typing 'nconvert' into their console. Check it! Unfortunately for these OS's there is no simple DXT5nm conversion program yet. However, I am planning to add a respective cross-platform output module to my next nmtools update!

    If you managed to follow me until here without getting hooked up ... Congratulations! You are now perfectly READY for attempting your own 64k MONSTER normalmap on your computer!!!

    3.6: Step-by-step recipe for the 'real thing'
    First of all, you need to download the 1.2 GB gz-packed input heightmap of Earth from the NASA Arctic server. Here is the link, ready for clicking.

    Remember, unlike our 4k warming-up input heightmap above, this file uses 'big endian' storage mode of its 16 bit integer height values. Also note that some heightmaps e.g. of Mars use 'little endian' storage mode, instead. Therefore, you have to definitely find out the storage mode from the docs before running the nmtools.

    In conclusion, for our present task, all 'little endian' machines (Windows, Linux, Intel Mac) need to insert the byteswap = 1 option in the first nmtool command.

    Note that you can always find out whether your machine is 'little endian' or 'big endian' by executing any nmtool, e.g. 'nms' without arguments from the console. Watch out for the output concerning your machine's endedness!

    Depending on your line, the download may take a little while. I use the "DownThemAll" extension for the Mozilla Firefox 1.5.0.x browser that gives me an effective download speed equal to my fast NOMINAL one of 750 KByte/sec = 6000 Kbit/sec! The download initiated with clicking 'dTaOneClick!' downloads 10 parts of the file in parallel and thus takes only ~25 mins in my case. The actual server speed is limited to a much lower bandwidth, as so often!

    Next follow some comments as to gz-unpacking that is required above.

    There are two gz-unpacking options:
    1. For Windows, you may use e.g. winRAR to unpack the 85k heightmap with a click, which however requires 8 GB of storage on your machine (instead of merely 1.2 GB for the original, gz-packed heightmap)! If you have lots of disk space and desire to minimize the use of the console, choose this option.
    2. This option is way more elegant and is certainly the method of choice at least for Linux and Mac-OSX users!

      The 'native' GNU gz-packing/unpacking comand-line tool, 'gzip', can be made to work with STDIN/STDOUT just like the nmtools. Hence you may simply place the gz-unpacking command in front of the first nmtool command and use the pipe operator ' | ' to feed the gzip output into the first nmtool command! This is particularly attractive, since gzip is STANDARD both in Linux and for MAC's. Hence only the Windows users must quickly install gzip with two clicks

    Next comes some small 'special' effort required for Windows users after installing 'gzip':

    Windows:
    To make the 'gzip' command-line tool known in any application directory, you must once more follow the manual setting procedure of the Path environment that I have described in great detail for 'nconvert' in the previous subsection 3.5 !

    Try it. I am convinced you will manage all by yourself this time! Remember, the final test of success is to execute 'gzip' from the console without arguments by typing 'gzip' at the command prompt. See if you get a help output. Then 'gzip' was properly recognized and you are 'in business'!

    Now it's time to impress you with the elegance of the pipe mechanism. Here would be the single command-line required to do a full set of 2048 level 5 normalmap tiles in PPM format:

    gzip -dc <srtm_ramp2.world.86400x43200.bin.gz | resc2pow2 86400 1 | nmstiles 5 6378.14 65536 2.5

    Note that this command uses the 1.2 GB gz-packed 85k input heightmap (in the current directory), just as it arrived from the download! No additional storage for the 8 GB unpacked monster is required!

    Follow this piped command by the simple format conversion command:

    nconvert -out png *.ppm

    ... And you have your 2048 level5 tiles in lossless PNG format ready before you managed to finish the next bottle of beer ;-)

    Now comes a more systematic step-by-step instruction about how you arrive at a complete set of tiles for level0 ...level5. Of course the following instructions can elegantly be arranged in a batch script, which I shall address a little later in this department ...

    Step-by-step instructions
    • Start off by creating a VT tile directory, named e.g. "SRTM_RAMP2_earth_nmap.1k". Next, create a number of subdirectories, level0 ...level5. Open the console and 'cd' to the level5 directory which also has to contain the original, downloaded 85k heightmap file, either still gz-packed [1.2 GB only] or already gz-unpacked [8GB!], according to your preference.

     

    Execute these commands in the level5 directory:


    For 'little endian' PC's with gz-unpacked heightmap file, type in the console:

    resc2pow2 86400 1 < srtm_ramp2.world.86400x43200.bin > srtm_ramp2.world.64k.bin

    Or for 'big endian' Macs with gz-unpacked heightmap file, type:

    resc2pow2 86400 < srtm_ramp2.world.86400x43200.bin > srtm_ramp2.world.64k.bin

    Or with installed 'gzip' for 'little endian' PC's, type:

    gzip -dc <srtm_ramp2.world.86400x43200.bin.gz | resc2pow2 86400 1 > srtm_ramp2.world.64k.bin

    Or with installed 'gzip' for 'big endian' Macs, type:

    gzip -dc <srtm_ramp2.world.86400x43200.bin.gz | resc2pow2 86400 > srtm_ramp2.world.64k.bin

    After these alternatives depending on your setup & machines, the rest is all the same for everyone:

    halfsize 65536 < srtm_ramp2.world.64k.bin > srtm_ramp2.world.32k.bin
    halfsize 32768 < srtm_ramp2.world.32k.bin > srtm_ramp2.world.16k.bin
    halfsize 16384 < srtm_ramp2.world.16k.bin > srtm_ramp2.world.8k.bin
    halfsize   8192 < srtm_ramp2.world.8k.bin > srtm_ramp2.world.4k.bin
    halfsize   4096 < srtm_ramp2.world.4k.bin > srtm_ramp2.world.2k.bin

    Move srtm_ramp2.world.32k.bin into the level4 directory;
    Move srtm_ramp2.world.16k.bin into the level3 directory;
    Move srtm_ramp2.world.8k.bin into the level2 directory;
    Move srtm_ramp2.world.4k.bin into the level1 directory;
    Move srtm_ramp2.world.2k.bin into the level0 directory.

    Execute these commands in the adequate level5...level0 directories as specified:


    Next, you type in the level5 directory:

    nmstiles 5 6378.14 65536 2.5 < srtm_ramp2.world.64k.bin

    When all 2048 tiles are generated (watch the counter), 'cd' to the level4 directory and type

    nmstiles 4 6378.14 32768 2.5 < srtm_ramp2.world.32k.bin

    When all 512 tiles are generated (watch the counter), 'cd' to the level3 directory and type

    nmstiles 3 6378.14 16384 2.5 < srtm_ramp2.world.16k.bin

    When all 128 tiles are generated (watch the counter), 'cd' to the level2 directory and type

    nmstiles 2 6378.14 8192 2.5 < srtm_ramp2.world.8k.bin

    When all 32 tiles are generated (watch the counter), 'cd' to the level1 directory and type

    nmstiles 1 6378.14 4096 2.5 < srtm_ramp2.world.4k.bin

    When all 8 tiles are generated (watch the counter), 'cd' to the level0 directory and type

    nmstiles 0 6378.14 2048 2.5 < srtm_ramp2.world.2k.bin

    That's almost it! Watch the output text during the ongoing calculations. Of course you may simply copy and paste these commands one by one from your browser into the console...

    All you need to do finally is to quickly convert the PPM tile format to PNG format, for example. With nconvert it's VERY fast (cf. Subsect. 3.5). Note, DXT3 format is by far not good enough for that resolution! For Windows and Celestia (pre x) 1.5.0 I can recommend DXT5nm conversion by means of NVIDIA's nvDXT tools, as described extensively in the previous Subsect. 3.5!

    • For PNG conversion, type in each of the level0..level5 sub-directories:

      span style="color: #dc143c; font-family: courier, monospace">nconvert -out png *.ppm
    • For DXT5nm conversion under Windows (only!), type in each of the level0..level5 sub-directories:

      nvdxt -quality_highest -file *.ppm -nomipmap -dxt5nm

      After nvdxt has finished in each sub-directory, rename all *.dds file output into *.dxt5nm, as required by Celestia 1.5.0. Type at the console prompt:

      ren *.dds *.dxt5nm

    Then you are done and ready for trying your achievement out in Celestia! Congratulations! Moreover you are now perfectly prepared to attempt making e.g. a normalmap for Mars in highest available resolution by means of thenmtools!

    This whole sequence of commands can conveniently be written into an executable shell script. The total execution of all the mentioned commands takes ONLY 15-20 minutes on my machine! Enjoy and good luck!



    4: An appropriate 64k BMNG base texture

    Forthcoming!

Statcounter