NeXus data files contain four types of entity: data groups, data fields, attributes, and links.

In fact, a NeXus file can be viewed as a computer file system. Just as files are stored in folders (or subdirectories) to make them easy to locate, so NeXus fields are stored in groups. The group hierarchy is designed to make it easy to navigate a NeXus file.

Example of a NeXus File

The following diagram shows an example of a NeXus file represented as a tree structure.

Figure 1.1. Example of a NeXus file

Note that each field is identified by a name, such as counts, but each group is identified both by a name and, after a colon as a delimiter, the class type, e.g., monitor:NXmonitor). The class types, which all begin with NX, define the sort of fields that the group should contain, in this case, counts from a beamline monitor. The hierarchical design, with data items nested in groups, makes it easy to identify information if you are browsing through a file.

Important Classes

Here are some of the important classes found in nearly all NeXus files. A complete list can be found in the NeXus Design chapter.

Note

Note that NXentry and NXdata are the only two classes necessary to store the minimum amount of information in a valid NeXus data file.

NXentry

Required: The top level of any NeXus file contains one or more groups with the class NXentry. These contain all the data that is required to describe an experimental run or scan. Each NXentry typically contains a number of groups describing sample information (class NXsample), instrument details (class NXinstrument), and monitor counts (class NXmonitor).

NXdata

Required: Each NXentry group contains one or more groups with class NXdata. These groups contain the experimental results in a self-contained way, i.e., it should be possible to generate a sensible plot of the data from the information contained in each NXdata group. That means it should contain the axis labels and titles as well as the data.

NXsample

A NXentry group will often contain a group with class NXsample. This group contains information pertaining to the sample, such as its chemical composition, mass, and environment variables (temperature, pressure, magnetic field, etc.).

NXinstrument

There might also be a group with class NXinstrument. This is designed to encapsulate all the instrumental information that might be relevant to a measurement, such as flight paths, collimations, chopper frequencies, etc.

Figure 1.2. NXinstrument excerpt

Since an instrument can comprise several beamline components each defined by several parameters, they are each specified by a separate group. This hides the complexity from generic file browsers, but makes the information available in an intuitively obvious way if it is required.

Simple Example

NeXus data files do not need to be complicated. In fact, the following diagram shows an extremely simple NeXus file (in fact, the simple example shows the minimum information necessary for a NeXus data file) that could be used to transfer data between programs. (Later in this section, we show how to write and read this simple example.)

Figure 1.3. Simple Example

This illustrates the fact that the structure of NeXus files is extremely flexible. It can accommodate very complex instrumental information, if required, but it can also be used to store very simple data sets. In the next example, a NeXus data file is shown as XML:

Example 1.1. verysimple.xml: A very simple NeXus Data file (in XML)

<?xml version="1.0" encoding="UTF-8"?>
  <NXroot NeXus_version="4.3.0" XML_version="mxml"
      file_name="verysimple.xml"
      xmlns="http://definition.nexusformat.org/schema/3.1"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://definition.nexusformat.org/schema/3.1 
                          http://definition.nexusformat.org/schema/3.1/BASE.xsd"
      file_time="2010-11-12T12:40:17-06:00">
    <NXentry name="entry">
      <NXdata name="data">
        <counts NAPItype="NX_INT64[15]" long_name="photodiode counts" signal="NX_INT32:1" axes="two_theta">
                              1193                     4474 
                             53220                   274310 
                            515430                   827880 
                           1227100                  1434640 
                           1330280                  1037070 
                            598720                   316460 
                             56677                     1000 
                              1000 
        </counts>
        <two_theta NAPItype="NX_FLOAT64[15]" units="degrees" long_name="two_theta (degrees)">
                  18.90940         18.90960         18.90980         18.91000 
                  18.91020         18.91040         18.91060         18.91080 
                  18.91100         18.91120         18.91140         18.91160 
                  18.91180         18.91200         18.91220 
        </two_theta>
      </NXdata>
    </NXentry>
  </NXroot>

NeXus files are easy to create. This example NeXus file was created using a short Python program and NeXpy:

Example 1.2. verysimple.py: Using NeXpy to write a very simple NeXus Data file (in HDF5)

#
# This example uses NeXpy to build the verysimple.nx5 data file.

from nexpy.api import nexus

angle = [18.9094, 18.9096, 18.9098, 18.91,  18.9102, 
         18.9104, 18.9106, 18.9108, 18.911, 18.9112, 
	 18.9114, 18.9116, 18.9118, 18.912, 18.9122]
diode = [1193, 4474, 53220, 274310, 515430, 827880, 
         1227100, 1434640, 1330280, 1037070, 598720, 
	 316460, 56677, 1000, 1000]

two_theta = nexus.SDS(angle, name="two_theta", 
               units="degrees", 
	       long_name="two_theta (degrees)")
counts = nexus.SDS(diode, name="counts", long_name="photodiode counts")
data = nexus.NXdata(counts,[two_theta])
data.nxsave("verysimple.nx5")

# The verysimple.xml file was built with this command:
#	nxconvert -x verysimple.nx5 verysimple.xml
# and then hand-edited (line breaks) for display.

If the design principles are followed, it will be easy for anyone browsing a NeXus file to understand what it contains, without any prior information. However, if you are writing specialized visualization or analysis software, you will need to know precisely what specific information is contained in advance. For that reason, NeXus provides a way of defining the format for particular instrument types, such as time-of-flight small angle neutron scattering. This requires some agreement by the relevant communities, but enables the development of much more portable software.

The set of data storage objects is divided into three parts: base classes, application definitions, and contributed definitions. The base classes represent a set of components that define the dictionary of all possible terms to be used with that component. The application definitions specify the minimum required information to satisfy a particular scientific or data analysis software interest. The contributed definitions have been submitted by the scientific community for incubation before they are adopted by the NIAC or for availability to the community.

These instrument definitions are formalized as XML files, using NXDL, (as described in the NXDL chapter in Volume II of this documentation) to specify the names of data fields, and other NeXus data objects. The following is an example of such a file for the simple NeXus file shown above.

<?xml version="1.0" ?> 
<definition 
  xmlns="http://definition.nexusformat.org/nxdl/3.1" 
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://definition.nexusformat.org/nxdl/3.1 ../nxdl.xsd"
  category="base"
  name="verysimple"
  version="1.0"
  svnid="$Id: verysimple.nxdl.xml 730 2010-11-12 18:40:01Z Pete Jemian $"
  type="group" extends="NXobject">
  
  <doc>
    A very simple NeXus NXDL file
  </doc>
  <group type="NXentry">
    <group type="NXdata">
      <field name="counts" type="NX_INT"  units="NX_UNITLESS">
        <doc>counts recorded by detector</doc>
      </field>
      <field name="two_theta" type="NX_FLOAT" units="NX_ANGLE">
        <doc>rotation angle of detector arm</doc>
      </field>
    </group>
  </group>
</definition>

For complete examples of reading and writing NeXus data files, refer to the Examples of reading or writing NeXus data files chapter in Volume II. This chapter has several examples of writing and reading NeXus data files. If you want to define the format of a particular type of NeXus file for your own use, e.g. as the standard output from a program, you are encouraged to publish the format using this XML format. An example of how to do this is shown in the section titled Creating a NXDL Specification.

NeXus data files are high-level so the user only needs to know how the data are referenced in the file but does not need to be concerned where the data are stored in the file. Thus, the data are most easily accessed using a subroutine library tuned to the specifics of the data format.

In the past, a data format was defined by a document describing the precise location of every item in the data file, either as row and column numbers in an ASCII file, or as record and byte numbers in a binary file. It is the job of the subroutine library to retrieve the data. This subroutine library is commonly called an application-programmer interface or API.

For example, in NeXus, a program to read in the wavelength of an experiment would contain lines similar to the following:

NXopendata (fileID, "wavelength");
NXgetdata (fileID, lambda);
NXclosedata (fileID);

In this example, the program requests the value of the data that has the label wavelength, storing the result in the variable lambda. fileID is a file identifier that is provided by NeXus when the file is opened.

We shall provide a more complete example when we have discussed the contents of the NeXus files.

NeXus began as a group of scientists with the goal of defining a common data storage format to exchange experimental results and to exchange ideas about how to analyze them.

The NeXus Scientific Community provides the scientific data, advice, and continued involvement with the NeXus standard. NeXus provides a forum for the scientific community to exchange ideas in data storage through the NeXus wiki.

The NeXus International Advisory Committee supervises the development and maintenance of the NeXus common data format for neutron, x-ray, and muon science. The NIAC: The NeXus International Advisory Committee supervises a technical committee to oversee the NeXus Application Programmer Interface (NAPI: The NeXus Application Programming Interface) and the NeXus class definitions.

An important motivation for the design of NeXus was to simplify the creation of a default plot view. While the best representation of a set of observations will vary, depending on various conditions, a good suggestion is often known a priori. This suggestion is described in the NXdata element so that any program that is used to browse NeXus data files can provide a best representation without request for user input.

Another important motivation for NeXus, indeed the raison d'etre, was the community need to analyze data from different user facilities. A single data format that is in use at a variety of facilities would provide a major benefit to the scientific community. This unified format should be capable of describing any type of data from the scientific experiments, at any step of the process from data acquisition to data reduction and analysis. This unified format also needs to allow data to be written to storage as efficiently as possible to enable use with high-speed data acquisition.

Self-description, combined with a reliance on a multi-platform (and thereby portable) data storage format, are valued components of a data storage format where the longevity of the data is expected to be longer than the lifetime of the facility at which it is acquired. As the name implies, self-description within data files is the practice where the structure of the information contained within the file is evident from the file itself. A multi-platform data storage format must faithfully represent the data identically on a variety of computer systems, regardless of the bit order or byte order or word size native to the computer.

The scientific community continues to grow the various types of data to be expressed in data files. This practice is expected to continue as part of the investigative process. To gain broad acceptance in the scientific user community, any data storage format proposed as a standard would need to be extendable and continue to provide a means to express the latest notions of scientific data.

The maintenance cost of common data structures meeting the motivations above (self-describing, portable, and extendable) is not insurmountable but is often well-beyond the research funding of individual members of the muon, neutron, and X-ray science communities. Since it is these members that drive the selection of a data storage format, it is necessary for the user cost to be as minimal as possible. In this case, experience has shown that the format must be in the public-domain for it to be commonly accepted as a standard. A benefit of the public-domain aspect is that the source code for the API is open and accessible, a point which has received notable comment in the scientific literature.

More recently, NeXus has recognized that part of the scientific community with a desire to write and record scientific data, has small data volumes and a large aversion to the requirement of a complicated API necessary to access data in binary files such as HDF. For such information, the NeXus API has been extended by the addition of the eXtensible Markup Language^[1] (XML) as an alternative to HDF. XML is a text-based format that supports compression and structured data and has broad usage in business and e-commerce. While possibly complicated, XML files are human readable, and tools for translation and extraction are plentiful. The API has routines to read and write XML data and to convert between HDF and XML.

NeXus as a Common Data Exchange Format

By the late 1980s, it had become common practice for a scientific instrument or facility to define its own data format, often at the convenience of the local computer system. Data from these facilities were not easily interchanged due to various differences in computer systems and the compression schemes of binary data. It was necessary to contact the facility to obtain a description so that one could write an import routine in software. Experience with facilities closing (and subsequent lack of access to information describing the facility data format) revealed a significant limitation with this common practice. Further, there existed a N * N number of conversion routines necessary to convert data between various formats. In Figure 1.4, “N separate file formats”, circles represent different data file formats while arrows represent conversion routines. Note that the red circle only maps to one other format.

Figure 1.4. N separate file formats

One early idea has been for NeXus to become the common data exchange format, and thereby reduce the number of data conversion routines from N * N down to 2N, as show in Figure 1.5, “N separate file formats joined by a common NeXus converter”.

Figure 1.5. N separate file formats joined by a common NeXus converter

A necessary feature of a standard for the interchange of scientific data is a defined dictionary (or lexicography) of terms. This dictionary declares the expected spelling and meaning of terms when they are present so that it is not necessary to search for all the variant forms of energy when it is used to describe data (e.g., E, e, keV, eV, nrg, ...).

NeXus recognized that each scientific specialty has developed a unique dictionary and needs to categorize data using those terms. The NeXus Application Definitions provide the means to document the lexicography for use in data files of that scientific specialty.

The NeXus Application Program Interface (API) provides a set of subroutines that make it easy to read and write NeXus files. These subroutines are available in C, Fortran 77, Fortran 90, Java, Python, C++, and IDL.

The API uses a very simple state model to navigate through a NeXus file. (Compare this example with NAPI Simple 2-D Write Example (C, F77, F90), in the NAPI chapter of Volume II, using the native HDF5 commands.) When you open a file, the API provides a file handle, which stores the current location, i.e. which group and/or field is currently open. Read and write operations then act on the currently open entity. Following the simple example of Simple example data file, we walk through a schematic of NeXus program written in C (without any error checking or real data).

#include "napi.h"

 int main()
 {
    NXhandle fileID;
    NXopen ("NXfile.nxs", NXACC_CREATE, &fileID);
      NXmakegroup (fileID, "Scan", "NXentry");
      NXopengroup (fileID, "Scan", "NXentry");
        NXmakegroup (fileID, "data", "NXdata");
        NXopengroup (fileID, "data", "NXdata");
        /* somehow, we already have arrays tth and counts, each length n*/
          NXmakedata (fileID, "two_theta", NX_FLOAT32, 1, &n);
          NXopendata (fileID, "two_theta");
            NXputdata (fileID, tth);
            NXputattr (fileID, "units", "degrees", 7, NX_CHAR);
          NXclosedata (fileID);  /* two_theta */
          NXmakedata (fileID, "counts", NX_FLOAT32, 1, &n);
          NXopendata (fileID, "counts");
            NXputdata (fileID, counts);
          NXclosedata (fileID);  /* counts */
        NXclosegroup (fileID);  /* data */
      NXclosegroup (fileID);  /* Scan */
    NXclose (&fileID);
    return;
}

Reading a NeXus file works in the same way by traversing the tree with the handle.

This schematic C code will read the two-theta array created in Writing a simple NeXus file using NAPI above. (Again, compare this example with one in the NAPI chapter of Volume II^[10] using the native HDF5 commands.)

 NXopen ('NXfile.nxs', NXACC_READ, &fileID);
   NXopengroup (fileID, "Scan", "NXentry");
     NXopengroup (fileID, "data", "NXdata");
       NXopendata (fileID, "two_theta");
         NXgetinfo (fileID, &rank, dims, &datatype);
         NXmalloc ((void **) &tth, rank, dims, datatype);
         NXgetdata (fileID, tth);
       NXclosedata (fileID);
     NXclosegroup (fileID);
   NXclosegroup (fileID);
 NXclose (fileID);

NeXus files can also be viewed by a command-line browser, nxbrowse, which is included as a helper tool in the NeXus API distribution. The following is an example session of using nxbrowse to view a data file. The following commands are used in Example 1.7, “Using nxbrowse” in this session:

%> nxbrowse lrcs3701.nxs

NXBrowse 3.0.0. Copyright (C) 2000 R. Osborn, M. Koennecke, P. Klosowski
    NeXus_version = 1.3.3
    file_name = lrcs3701.nxs
    file_time = 2001-02-11 00:02:35-0600
    user = EAG/RO
NX> dir
  NX Group : Histogram1 (NXentry)
  NX Group : Histogram2 (NXentry)
NX> open Histogram1
NX/Histogram1> dir
  NX Data  : title[44] (NX_CHAR)
  NX Data  : analysis[7] (NX_CHAR)
  NX Data  : start_time[24] (NX_CHAR)
  NX Data  : end_time[24] (NX_CHAR)
  NX Data  : run_number (NX_INT32)
  NX Group : sample (NXsample)
  NX Group : LRMECS (NXinstrument)
  NX Group : monitor1 (NXmonitor)
  NX Group : monitor2 (NXmonitor)
  NX Group : data (NXdata)
NX/Histogram1> read title
  title[44] (NX_CHAR) = MgB2 PDOS 43.37g 8K 120meV E0@240Hz T0@120Hz
NX/Histogram1> open data
NX/Histogram1/data> dir
  NX Data  : title[44] (NX_CHAR)
  NX Data  : data[148,750] (NX_INT32)
  NX Data  : time_of_flight[751] (NX_FLOAT32)
  NX Data  : polar_angle[148] (NX_FLOAT32)
NX/Histogram1/data> read time_of_flight
  time_of_flight[751] (NX_FLOAT32) = [ 1900.000000 1902.000000 1904.000000 ...]
    units = microseconds
    long_name = Time-of-Flight [microseconds]
NX/Histogram1/data> read data
  data[148,750] (NX_INT32) = [ 1 1 0 ...]
    units = counts
    signal = 1 
    long_name = Neutron Counts
    axes = polar_angle:time_of_flight
NX/Histogram1/data> close
NX/Histogram1> close
NX> quit

The source code of nxbrowse^[11] provides an example of how to write a NeXus reader. The test programs included in the NeXus API may also be useful to study.

NeXus files consist of data groups, which contain fields and/or other groups to form a hierarchical structure. This hierarchy is designed to make it easy to navigate a NeXus file by storing related fields together. Data groups are identified both by a name, which must be unique within a particular group, and a class. There can be multiple groups with the same class but they must have different names (based on the HDF rules). For the class names used with NeXus data groups the prefix NX is reserved. Thus all NeXus class names start with NX.

Data fields contain the essential information stored in a NeXus file. They can be scalar values or multidimensional arrays of a variety of sizes (1-byte, 2-byte, 4-byte, 8-byte) and types (integers, floats, characters). The fields may store both experimental results (counts, detector angles, etc), and other information associated with the experiment (start and end times, user names, etc). Data fields are identified by their names, which must be unique within the group in which they are stored.

Attributes are extra (meta-)information that are associated with particular fields. They are used to annotate the data, e.g. with physical units or calibration offsets, and may be scalar numbers or character strings. In addition, NeXus uses attributes to identify plottable data and their axes, etc. A description of possible attributes can be found in table data attributes. Finally, NeXus files themselves have global attributes which are listed in the global attributes table. that identify the NeXus version, file creation time, etc. Attributes are identified by their names, which must be unique in each field.

Links are pointers to existing data somewhere else. The concept is very much like symbolic links in a unix filesystem. The NeXus definition sometimes requires to have access to the same data in different groups in the same file. For example: detector data is stored in the NXinstrument/NXdetector group but may be needed in NXdata for automatic plotting. Rather then replicating the data, NeXus uses links in such situations. See the figure for a more descriptive representation of the concept of linking.

Figure 2.1. Linking in a NeXus file

Data groups often describe objects in the experiment (monitors, detectors, monochromators, etc.), so that the contents (both data fields and/or other data groups) comprise the properties of that object. NeXus has defined a set of standard objects, or base classes, out of which a NeXus file can be constructed. This is each data group is identified by a name and a class. The group class, defines the type of object and the properties that it can contain, whereas the group name defines a unique instance of that class. These classes are defined in XML using the NeXus Definition Language (NXDL) format. All NeXus class types adopted by the NIAC must begin with NX. Classes not adopted by the NIAC must not start with NX.

Not all classes define physical objects. Some refer to logical groupings of experimental information, such as plottable data, sample environment logs, beam profiles, etc. There can be multiple instances of each class. On the other hand, a typical NeXus file will only contain a small subset of the possible classes.

NeXus base classes are not proper classes in the same sense as used in object oriented programming languages. In fact the use of the term classes is actually misleading but has established itself during the development of NeXus. NeXus base classes are rather dictionaries of field names and their meanings which are permitted in a particular NeXus group implementing the NeXus class. This sounds complicated but becomes easy if you consider that most NeXus groups describe instrument components. Then for example, a NXmonochromator base class describes all the possible field names which NeXus allows to be used to describe a monochromator.

Most NeXus base classes represent instrument components. Some are used as containers to structure information in a file (NXentry, NXcollection, NXinstrument, NXprocess, NXparameter). But there are some base classes which have special uses which need to be mentioned here:

These groups can appear anywhere in the NeXus hierarchy, where needed. Preferably close to the component they annotate or in a NXcollection. All of the base classes are documented in the reference manual.

The objects described so far provide us with the means to store data from a wide variety of instruments, simulations or processed data as resulting from data analysis. But NeXus strives to express strict standards for certain applications of NeXus too. The tool which NeXus uses for the expression of such strict standards is the NeXus Application Definition. A NeXus Application Definition describes which groups and data items have to be present in a file in order to properly describe an application of NeXus. For example for describing a powder diffraction experiment. Typically an application definition will contain only a small subset of the many groups and fields defined in NeXus. NeXus application definitions are also expressed in the NeXus Definition Language (NXDL). A tool exists which allows one to validate a NeXus file against a given application definition.

Another way to look at a NeXus application definition is as a contract between a file producer (writer) and a file consumer (reader). The contract reads: If you write your files following a particular NeXus application definition, I can process these files with my software.

Yet another way to look at a NeXus application definition is to understand it as an interface definition between data files and the software which uses this file. Much like an interface in the Java or other modern object oriented programming languages.

In contrast to NeXus base classes, NeXus supports inheritance in application definitions.

Please note that a NeXus Application Definition will only define the bare minimum of data necessary to perform common analysis with data. Practical files will nearly always contain more data. One of the beauties of NeXus is that it is alwasy possible to add more data to a file without breaking its compliance with its application definition.

As stated above, NeXus uses the McStas coordinate system as its laboratory coordinate system. The instrument is given a global, absolute coordinate system where the z axis points in the direction of the incident beam, the x axis is perpendicular to the beam in the horizontal plane pointing left as seen from the source, and the y axis points upwards. See below for a drawing of the McStas coordinate system. The origin of this coordinate system is the sample position or, if this is ambiguous, the center of the sample holder with all angles and translations set to zero. The McStas coordinate system is illustrated in figure McStas Coordinate System.

Figure 2.2. The McStas Coordinate System

The NeXus NXgeometry class directly uses the McStas coordinate system. NXgeometry classes can appear in any component in order to specify its position. The suggested name to use is geometry. In NXgeometry the NXtranslation/values field defines the absolute position of the component in the McStas coordinate system. The NXorientation/value field describes the orientation of the component as a vector of in the McStas coordinate system.

In this system, the instrument is considered as a set of components through which the incident beam passes. The variable distance is assigned to each component and represents the effective beam flight path length between this component and the sample. A sign convention is used where negative numbers represent components pre-sample and positive numbers components post-sample. At each component there is local spherical coordinate system with the angles polar_angle and azimuthal_angle. The size of the sphere is the distance to the previous component.

In order to understand this spherical polar coordinate system it is helpful to look initially at the common condition that azimuthal_angle is zero. This corresponds to working directly in the horizontal scattering plane of the instrument. In this case polar_angle maps directly to the setting commonly known as two theta. Now, there are instruments where components live outside of the scattering plane. Most notably detectors. In order to describe such components we first apply the tilt out of the horizontal scattering plane as the azimuthal_angle. Then, in this tilted plane, we rotate to the component. The beauty of this is that polar_angle is always two theta. Which, in the case of a component out of the horizontal scattering plane, is not identical to the value read from the motor responsible for rotating the component. This situation is shown in Figure: Polar Coordinate System.

Figure 2.3.  NeXus Simple (Spherical Polar) Coordinate System

Another way to look at coordinates is through the use of transformation matrices. In this world view, the absolute position of a component or a detector pixel with respect to the laboratory corrdinate system is calculated by applying a series of translations and rotations. These operations are commonly expressed as transformation matrices and their combination as matrix multiplication. A very important aspect is that the order of application of the individual operations does matter. Another important aspect is that any operation transforms the whole coordinate system and gives rise to a new local coordinate system. The mathematics behind this is well known and used in such applications such as industrial robot control, space flight and computer games. The beauty in this comes from the fact that the operations to apply map easily to instrument settings and constants. It is also easy to analyze the contribution of each individual operation: this can be studied under the condition that all other operations are at a zero setting.

In order to use coordinate transformations, several morsels of information need to be known:

The type and direction of the NeXus standard operations is documented in Table 2.3, “ Actions of standard NeXus fields ”. NeXus can now also allow non standard operations to be stored in data files. In such cases additional data attributes are required which describe the operation. These are transformation_type which can be either translation or rotation. The other is vector which is 3 float values describing the direction of translation or rotation. The value is of course always the value of the data field in the data file.

How NeXus describes the order of operations to apply has not yet been decided upon. The authors favourite scheme is to use a special field at each instrument component, named transform which describes the operations to apply to get the component into its position as a list of colon separated paths to the operations to apply relative to the current NXentry. For paths in the same group, only the name need to be given. Detectors may need two such fields: the transfrom field to get the get the detector as a whole into its position and a transform_pixel field which describes how the absolute position of a detector pixel can be calculated.

For the NeXus spherical coordinate system, the order is implicit and is given in the next example.

azimuthal_angle:polar_angle:distance

This is also a nice example of the application of transformation matrices:

An example raw data hierarchy is shown in figure Raw Data (only showing the relevant parts of the data hierarchy). In the example shown, the data field in the NXdata group is linked to the 2-D detector data (a 512x512 array of 32-bit integers) which has the attribute signal=1. Note that [,] represents a 2D array.

	entry:NXentry
		instrument:NXinstrument
			source:NXsource
			....
			detector:NXdetector
				data:NX_INT32[512,512]
					@signal = 1
		sample:NXsample
		control:NXmonitor
		data:NXdata
			data --> /entry/instrument/detector/data

An NXentry describing raw data contains at least a NXsample, one NXmonitor, one NXdata and a NXinstrument group. It is good practice to use the names sample for the NXsample group, control for the NXmonitor group holding the experiment controlling monitor and instrument for the NXinstrument group. The NXinstrument group contains further groups describing the individual components of the instrument as appropriate.

The NXdata group contains links to all those data items in the NXentry hierarchy which are required to put up a default plot of the data. As an example consider a SAXS instrument with a 2D detector. The NXdata will then hold a link to the detector image. If there is only one NXdata group, it is good practice to name it data. Otherwise, the name of the detector bank represented is a good selection.

Processed data, see figure Processed Data, in this context means the results of a data reduction or data analysis program. Note that [] represents a 1D array.

	entry:NXentry
		reduction:NXprocess
			program_name = "pyDataProc2010"
			version = "1.0a"
			input:NXparameter
				filename = "sn2013287.nxs"
		sample:NXsample
		data:NXdata
			data
				@signal = 1

NeXus stores such data in a simplified NXentry structure. A processed data NXentry has at minimum a NXsample, a NXdata and a NXprocess group. Again the preferred name for the NXsample group is sample. In the case of processed data, the NXdata group holds the result of the processing together with the associated axis data. The NXprocess group holds the name and version of the program used for this processing step and further NXparameter groups. These groups ought to contain the parameters used for this data processing step in suitable detail so that the processing step can be reproduced.

Optionally a processed data NXentry can hold a NXinstrument group with further groups holding relevant information about the instrument. The preferred name is again instrument. Whereas for a raw data file, NeXus strives to capture as much data as possible, a NXinstrument group for processed data may contain a much-reduced subset.

Especially at synchrotron facilities, there are experiments which perform several different methods on the sample at the same time. For example, combine a powder diffraction experiment with XAS. This may happen in the same scan, so the data needs to be grouped together. A suitable NXentry would need to adhere to two different application definitions. This leads to name clashes which cannot be easily resolved. In order to solve this issue, the following scheme was implemented in NeXus:

See figure NeXus Multi Method Hierarchy for an example hierarchy. Note that [,] represents a 2D array.

	entry:NXentry
		user:NXuser
		sample:NXsample
		instrument:NXinstument
			SASdet:NXdetector
				data:[,]
					@signal = 1
			fluordet:NXdetector
				data:[,]
					@signal = 1
			large_area:NXdetector
				data:[,]
		SAS:NXsubentry
			definition = "NXsas"
			instrument:NXinstrument
				detector:NXdetector
					data --> /entry/instrument/SASdet/data
			data:NXdata
				data --> /entry/instrument/SASdet/data
		Fluo:NXsubentry
			definition = "NXFluo"
			instrument:NXinstrument
				detector --> /entry/instrument/fluordet/data
				detector2 --> /entry/instrument/large_area/data
			data:NXdata
				detector --> /entry/instrument/fluordet/data

Group and field Names used within NeXus follow a naming convention which is made up from the following rules: The names of NeXus groups and fields must only contain a restricted set of characters. This set may be described by this regular expression syntax regular expression syntax:

[A-Za-z_][\w_]*

Note that this name pattern starts with a letter (upper or lower case) or "_" (underscore), then letters, numbers, and "_" and is limited to no more than 63 characters (imposed by the HDF5 rules for names).

Sometimes it is necessary to combine words in order to build a descriptive name for a data field or a group. In such cases lowercase words are connected by underscores.

number_of_lenses

For all data fields, only names from the NeXus base class dictionaries are to be used.^[13] If a data field name or even a complete component is missing, please suggest the addition to the NIAC. The addition will usually be accepted provided it is not a duplication of an existing field and adequately documented.

NeXus stores multi dimensional arrays of physical values in C language storage order, where the last dimension is the fastest varying. This is the rule. Good reasons are required to deviate from this rule.

It is possible to store data in storage orders other than C language order. As well it is possible to specify that the data needs to be converted first before being useful. Consider one situation, when data must be streamed to disk as fast as possible and conversion to C language storage order causes unnecessary latency. This case presents a good reason to make an exception to the standard rule.

   * raw data = 0 1 2 3 4 5 6 7 8 9
      size[1] = { 10 }  // assume uniform overall array dimensions

   * default stride:
      stride[1] = { 1 }
      offset[1] = { 0 }
      for i:
         result[i]:
            0 1 2 3 4 5 6 7 8 9

   * reverse stride:
      stride[1] = { -1 }
      offset[1] = { 9 }
      for i:
         result[i]:
            9 8 7 6 5 4 3 2 1 0

   * raw data = 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
      size[2] = { 4, 5 }  // assume uniform overall array dimensions

   * row major (C) stride:
      stride[2] = { 5, 1 }
      offset[2] = { 0, 0 }
      for i:
         for j:
            result[i][j]:
               0 1 2 3 4
               5 6 7 8 9
               10 11 12 13 14
               15 16 17 18 19

   * column major (Fortran) stride:
      stride[2] = { 1, 4 }
      offset[2] = { 0, 0 }
      for i:
         for j:
            result[i][j]:
               0 4 8 12 16
               1 5 9 13 17
               2 6 10 14 18
               3 7 11 15 19

   * "crazy reverse" row major (C) stride:
      stride[2] = { -5, -1 }
      offset[2] = { 4, 5 }
      for i:
         for j:
            result[i][j]:
               19 18 17 16 15
               14 13 12 11 10
               9 8 7 6 5
               4 3 2 1 0   

  * raw data = 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
         20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
         40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59
      size[3] = { 3, 4, 5 }  // assume uniform overall array dimensions

   * row major (C) stride:
      stride[3] = { 20, 5, 1 }
      offset[3] = { 0, 0, 0 }
      for i:
         for j:
            for k:
               result[i][j][k]:
                  0 1 2 3 4
                  5 6 7 8 9
                  10 11 12 13 14
                  15 16 17 18 19

                  20 21 22 23 24
                  25 26 27 28 29
                  30 31 32 33 34
                  35 36 37 38 39

                  40 41 42 43 44
                  45 46 47 48 49
                  50 51 52 53 54
                  55 56 57 58 59

   * column major (Fortran) stride:
      stride[3] = { 1, 3, 12 }
      offset[3] = { 0, 0, 0 }
      for i:
         for j:
            for k:
               result[i][j][k]:
                  0 12 24 36 48
                  3 15 27 39 51
                  6 18 30 42 54
                  9 21 33 45 57

                  1 13 25 37 49
                  4 16 28 40 52
                  7 19 31 43 55
                  10 22 34 46 58

                  2 14 26 38 50
                  5 17 29 41 53
                  8 20 32 44 56
                  11 23 35 47 59

NeXus supports numeric data as either integer or floating-point numbers. A number follows that indicates the number of bits in the word. The table above shows the regular expressions that matches the data type specifier.

%Y-%m-%dT%H:%M:%S%z

Given the plethora of possible applications of NeXus, it is difficult to define units to use. Therefore, the general rule is that you are free to store data in any unit you find fit. However, any data field must have a units attribute which describes the units, Wherever possible, SI units are preferred. NeXus units are written as a string attribute (NX_CHAR) and describe the engineering units. The string should be appropriate for the value. Values for the NeXus units must be specified in a format compatible with Unidata UDunits ^[15] Application definitions may specify units to be used for fields using an enumeration.

NeXus allows to store multi dimensional arrays of data. In most cases it is not sufficient to just have the indices into the array as a label for the dimensions of the data. Usually the information which physical value corresponds to an index into a dimension of the multi dimensional data set. To this purpose a means is needed to locate appropriate data arrays which describe what each dimension of a multi dimensional data set actually corresponds too. There is a standard HDF facility to do this: it is called dimension scales. Unfortunately, at a time, there was only one global namespace for dimension scales. Thus NeXus had to come up with its own scheme for locating axis data which is described here. A side effect of the NeXus scheme is that it is possible to have multiple mappings of a given dimension to physical data. For example a TOF data set can have the TOF dimension as raw TOF or as energy.

There are two methods of linking each data dimension to its respective dimension scale. The preferred method uses the axes attribute to specify the names of each dimension scale. The original method uses the axis attribute to identify with an integer the axis whose value is the number of the dimension. After describing each of these methods, the two methods will be compared. A prerequisite for both methods is that the data fields describing the axis are stored together with the multi dimensional data set whose axes need to be defined in the same NeXus group. If this leads to data duplication, use links.

  data:NXdata
    time_of_flight = 1500.0 1502.0 1504.0 ...
    polar_angle = 15.0 15.6 16.2 ...
    some_other_angle = 0.0 0.0 2.0 ...
    data = 5 7 14 ...
      @axes = polar_angle:time_of_flight
      @signal = 1

  data:NXdata
    time_of_flight = 1500.0 1502.0 1504.0 ...
      @axis = 1
      @primary = 1
    polar_angle = 15.0 15.6 16.2 ...
      @axis = 2
      @primary = 1
    some_other_angle = 0.0 0.0 2.0 ...
      @axis = 1
    data = 5 7 14 ...
      @signal = 1

There are very different types of detectors out there. Storing their data can be a challenge. As a general guide line: if the detector has some well defined form, this should be reflected in the data file. A linear detector becomes a linear array, a rectangular detector becomes an array of size xsize times ysize. Some detectors are so irregular that this does not work. Then the detector data is stored as a linear array, with the index being detector number till ndet. Such detectors must be accompanied by further arrays of length ndet which give azimuthal_angle, polar_angle and distance for each detector.

If data from a time of flight (TOF) instrument must be described, then the TOF dimension becomes the last dimension, for example an area detector of xsize vs. ysize is stored with TOF as an array with dimensions xsize, ysize, ntof.

Monitors , detectors that measure the properties of the experimental probe rather than the sample, have a special place in NeXus files. Monitors are crucial to normalize data. To emphasize their role, monitors are not stored in the NXinstrument hierarchy but on NXentry level in their own groups as there might be multiple monitors. Of special importance is the monitor in a group called control. This is the main monitor against which the data has to be normalized. This group also contains the counting control information, i.e. counting mode, times, etc.

Monitor data may be multidimensional. Good examples are scan monitors where a monitor value per scan point is expected or time-of-flight monitors.

Any program whose aim is to identify plottable data should use the following procedure:

Consult the NeXus API section, which describes the routines available to program these operations. In the course of time, generic NeXus browsers will provide this functionality automatically.

At its beginnings, the founders of NeXus identified the Hierarchical Data Format (HDF), initially from the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign (UIUC) and later spun off into its own group called The HDF Group (THG),^[16] as a multi-platform data storage format with capacity for conveying large data payloads and a substantial user community. HDF (now HDF5) was provided with software to read and write data (this is the application-programmer interface, or API) using a large number of computing systems in common use for neutron and X-ray science. HDF is a binary data file format that supports compression and structured data.

NeXus data structures map directly to HDF structures. NeXus groups are HDF4 vgroups or HDF5 groups, NeXus data sets (or fields) are HDF4 SDS (scientific data sets) or HDF5 datasets. Attributes map directly to HDF group or dataset attributes.

The only special case is the NeXus class name. HDF4 supports a group class which is set with the Vsetclass() call and read with VGetclass(). HDF-5 has no group class. Thus the NeXus class is stored as an attribute to the HDF-5 group with the name NX_class and value of the NeXus class name.

A NeXus link directly maps to the HDF linking mechanisms.

Perhaps the easiest way to view the implementation of NeXus in HDF5 is to view how the data structures look. For this, we use the h5dump command-line utility provided with the HDF5 support libraries. Short examples are provided for the basic NeXus data components:

See the sections NAPI Simple 2-D Write Example (C, F77, F90) and NAPI Simple 3-D Write Example (Python) in the Examples of reading or writing NeXus data files chapter of Volume II for examples that use the native HDF5 calls to write NeXus data files.

GROUP "entry" {
  ATTRIBUTE "NX_class" {
	 DATATYPE  H5T_STRING {
		   STRSIZE 7;
		   STRPAD H5T_STR_NULLPAD;
		   CSET H5T_CSET_ASCII;
		   CTYPE H5T_C_S1;
		}
	 DATASPACE  SCALAR
	 DATA {
	 (0): "NXentry"
	 }
  }
  # ... group contents
}

 DATASET "two_theta" {
	DATATYPE  H5T_IEEE_F64LE
	DATASPACE  SIMPLE { ( 31 ) / ( 31 ) }
	DATA {
	(0): 17.9261, 17.9259, 17.9258, 17.9256, 17.9254, 17.9252,
	(6): 17.9251, 17.9249, 17.9247, 17.9246, 17.9244, 17.9243,
	(12): 17.9241, 17.9239, 17.9237, 17.9236, 17.9234, 17.9232,
	(18): 17.9231, 17.9229, 17.9228, 17.9226, 17.9224, 17.9222,
	(24): 17.9221, 17.9219, 17.9217, 17.9216, 17.9214, 17.9213,
	(30): 17.9211
	}
	ATTRIBUTE "units" {
	   DATATYPE  H5T_STRING {
			 STRSIZE 7;
			 STRPAD H5T_STR_NULLPAD;
			 CSET H5T_CSET_ASCII;
			 CTYPE H5T_C_S1;
		  }
	   DATASPACE  SCALAR
	   DATA {
	   (0): "degrees"
	   }
	}
	# ... other attributes
 }

ATTRIBUTE "axes" {
   DATATYPE  H5T_STRING {
		 STRSIZE 9;
		 STRPAD H5T_STR_NULLPAD;
		 CSET H5T_CSET_ASCII;
		 CTYPE H5T_C_S1;
	  }
   DATASPACE  SCALAR
   DATA {
   (0): "two_theta"
   }
}

# NeXus links have two parts in HDF5 files.

# The dataset is created in some group.
# A "target" attribute is added to indicate the HDF5 path to this dataset.

ATTRIBUTE "target" {
   DATATYPE  H5T_STRING {
		 STRSIZE 21;
		 STRPAD H5T_STR_NULLPAD;
		 CSET H5T_CSET_ASCII;
		 CTYPE H5T_C_S1;
	  }
   DATASPACE  SCALAR
   DATA {
   (0): "/entry/data/two_theta"
   }
}

# then, the hard link is created that refers to the original dataset
# (Since the name is "two_theta" in this example, it is understood that
# this link is created in a different HDF5 group than "/entry/data".)

DATASET "two_theta" {
   HARDLINK "/entry/data/two_theta"
}

This takes a bit more work than HDF. At the root of NeXus XML file is a XML element with the name NXroot. Further XML attributes to NXroot define the NeXus file level attributes. An example NeXus XML data file is provided in the Introduction chapter as Example Example 1.1, “verysimple.xml: A very simple NeXus Data file (in XML)”

NeXus groups are encoded into XML as elements with the name of the NeXus class and an XML attribute name which defines the NeXus name of the group. Further group attributes become XML attributes. An example:

	<NXentry name="entry">
	</NXentry>

NeXus data sets are encoded as XML elements with the name of the data. An attribute NAPItype defines the type and dimensions of the data. The actual data is stored as PCDATA^[17] in the element. Another example:

	<mode NAPItype="NX_CHAR[7]">
		monitor
	</mode>
	<counts NAPItype="NX_INT32[4]">
		21 456  127876 319
	</counts>

Data are printed in appropriate formats and in C storage order. The codes understood for NAPItype are all the NeXus data type names. The dimensions are given in square brackets as a comma separated list. No dimensions need to be given if the data is just a single value. Data attributes are represented as XML attributes. If the attribute is not a text string, then the attribute is given in the form: type:value, for example: signal="NX_INT32:1".

NeXus links are stored in XML as XML elements with the name NAPIlink and a XML attribute target which stores the path to the linked entity in the file. If the item is linked under a different name, then this name is specified as a XML attribute name to the element NAPIlink.

The authors of the NeXus API worked with the author of the miniXML XML library to create a reasonably efficient way of handling numeric data with XML. Using the NeXus API handling something like 400 detectors versus 2000 time channels in XML is not a problem. But you may hit limits with XML as the file format when data becomes to large or you try to process NeXus XML files with general XML tools. General XML tools are normally ill prepared to process large amounts of numbers.

NeXus makes use of some special attributes for its internal purposes. These attributes are stored as normal group or data set attributes in the respective file format. These are:

Now the various groups of this empty NeXus file shell need to be filled. The next step is to look at a design drawing of WONI. Identify all the instrument components like collimators, detectors, monochromators etc. For each component decide which values need to be stored. As NeXus aims to describe the experiment as good as possible, strive to capture as much information as practical.

With the list of parameters to store for each component, consult the reference manual section on the NeXus base classes. You will find that for each of your instruments components there will be a suitable NeXus base class. Add this base class together with a name as a group under NXinstrument in your NeXus file hierarchy. Then consult the possible parameter names in the NeXus base class and match them with the parameters you wish to store for your instruments components.

As an example, consider the monochromator. You may wish to store: the wavelength, the d-value of the reflection used, the type of the monochromator and its angle towards the incoming beam. The reference manual tells you that NXcrystal is the right base class to use. Suitable fields for your parameters can be found in there to. After adding them to the basic NeXus file the file looks like in figure Figure 3.4, “Basic structure of a NeXus file with a monochromator added”

          entry:NXentry
    NXdata
    NXinstrument
       monochromator:Nxcrystal
          wavelength
          d_spacing
          rotation_angle
          reflection
          type
    NXmonitor
    NXsample

      

If a parameter or even a whole group is missing in order to describe your experiment, do not despair! Contact the NIAC and suggest to add the group or parameter. Give a little documentation what it is for. The NIAC will check that your suggestion is no duplicate and sufficiently documented and will then proceed to enhance the base classes with your suggestion.

A more elaborate example of the mapping process is given in the section Creating a NXDL Specification.

The NXdata/ group is supposed to contain the data required to put up a quick plot. For WONI this is a plot of counts versus two theta (polar_angle in NeXus) as can be seen in Figure 3.2, “Example Powder Diffraction Plot from (fictional) WONI at HYNES”. Now, in NXdata, create links to the appropriate data items in the NXinstrument hierarchy. In the case of WONI, both parameters live in the detector:NXdetector group.

Look at the section on NXsample in the NeXus reference manual. Choose appropriate parameters to store for your samples. Probably at least the name will be needed.

In order to normalize various experimental runs against each other it is necessary to know about the counting conditions and especially the monitor counts of the monitor used for normalization. The NeXus convention is to store such information in a control:NXmonitor group at NXentry level. Consult the reference for NXmonitor for field names. If additional monitors exist within your experiment, they will be stored as additional NXmonitor groups at entry level.

Consult the documentation for NXentry in order to find out under which names to store information such as titles, user names, experiment times etc.

A more elaborate example of this process can be found in the following section on creating an application definition.

With all this introductory stuff out of the way, let us look at the process required to define an application definition:

This is actually the hard bit. There are two things to consider:

For the first part, one of the NeXus guiding principles gives us - Guidance! “A NeXus file must contain all the data necessary for standard data analysis.”

Not more and not less for an application definition. Of course the definition of standard data for analysis or a standard plot depends on the science and the type of data being described. Consult senior scientists in the field about this is if you are unsure. Perhaps you must call an international meeting with domain experts to haggle that out. When considering this, people tend to put in everything which might come up. This is not the way to go.

A key test question is: Is this data item necessary for common data analysis? Only these necessary data items belong in an application definition.

The purpose of an application definition is that an author of upstream software who consumes the file can expect certain data items to be there at well defined places. On the other hand if there is a development in your field which analyzes data in a novel way and requires more data to do it, then it is better to err towards the side of more data.

Now for the case of WONI, the standard data analysis is either Rietveld refinement or profile analysis. For both purposes, the kind of radiation used to probe the sample (for WONI, neutrons), the wavelength of the radiation, the monitor (which tells us how long we counted) used to normalize the data, the counts and the two theta angle of each detector element are all required. Usually, it is desirable to know what is being analyzed, so some metadata would be nice: a title, the sample name and the sample temperature. The data typically being plotted is two theta against counts, as shown in Figure 3.2, “Example Powder Diffraction Plot from (fictional) WONI at HYNES” above. Summarizing, the basic information required from WONI is given in Table 3.1, “basic information required from WONI”.

If you start to worry that this is too little information, hold on, the section on Using an Application Definition (the section called “Using an Application Definition”) will reveal the secret how to go from an application definition to a practical file.

This step is actually easier then the first one. We need to map the data items which were collected in Step 1 into the NeXus hierarchy. A NeXus file hierarchy starts with an NXentry group. At this stage it is advisable to pull up the base class definition for NXentry and study it. The first thing you might notice is that NXentry contains a field named title. Reading the documentation, you quickly realize that this is a good place to store our title. So the first mapping has been found.

title = /NXentry/title

Another thing to notice in the NXentry base class is the existence of a group of class NXsample. This looks like a great place to store information about the sample. Studying the NXsample base class confirms this view and there are two new mappings:

sample name = /NXentry/NXsample/name
sample temperature = /NXentry/NXsample/temperature

Scanning the NXentry base class further reveals there can be a NXmonitor group at this level. Looking up the base class for NXmonitor reveals that this is the place to store our monitor information.

monitor = /NXentry/NXmonitor/data

For the other data items, there seem to be no solutions in NXentry. But each of these data items describe the instrument in more detail. NeXus stores instrument descriptions in the /NXentry/NXinstrument branch of the hierarchy. Thus, we continue by looking at the definition of the NXinstrument base class. In there we find further groups for all possible instrument components. Looking at the schematic of WONI (Figure 3.1, “The (fictional) WONI example powder diffractometer”), we realize that there is a source, a monochromator and a detector. Suitable groups can be found for these components in NXinstrument and further inspection of the appropriate base classes reveals the following further mappings:

probe = /NXentry/NXinstrument/NXsource/probe
wavelength = /NXentry/NXinstrument/NXcrystal/wavelength
two theta of detector elements = /NXentry/NXinstrument/NXdetector/polar angle
counts for each detector element = /NXentry/NXinstrument/NXdetector/data

Thus we mapped all our data items into the NeXus hierarchy! What still needs to be done is to decide upon the content of the NXdata group in NXentry. This group describes the data necessary to make a quick plot of the data. For WONI this is counts versus two theta. Thus we add this mapping:

two theta of detector elements = /NXentry/NXdata/polar angle
counts for each detector element = /NXentry/NXdata/data

The full mapping of WONI data into NeXus is documented in Table 3.2, “Full mapping of WONI data into NeXus”.

Looking at this one might get concerned that the two theta and counts data is stored in two places and thus duplicated. Stop worrying, this problem is solved at the NeXus API level. Typically NXdata will only hold links to the corresponding data items in /NXentry/NXinstrument/NXdetector.

In this step problems might occur. The first is that the base class definitions contain a bewildering number of parameters. This is on purpose: the base classes serve as dictionaries which define names for everything which possibly can occur. You do not have to give all that information. The key question is, as already said, What is required for typical data analysis for this type of application? You might also be unsure how to correctly store a particular data item. In such a case, contact the NIAC for help. Another problem which can occur is that you require to store information for which there is no name in one of the existing base classes or you have a new instrument component for which there is no base class alltogether. In such a case, please feel free to contact the NIAC with a suggestion for an extension of the base classes in question.

This is even easier. Some XML editing is necessary. Fire up your XML editor of choice and open a file. If your XML editor supports XML schema while editing XML, it is worth to load nxdl.xsd. Now your XML editor can help you to create a proper NXDL file. As always, the start is an empty template file. This looks like Example 3.1, “NXDL template file”. This is just the basic XML for a NXDL definition. It is advisable to change some of the documentation strings.

<?xml version="1.0" encoding="UTF-8"?>
<!--
# NeXus - Neutron and X-ray Common Data Format
# 
# Copyright (C) 2008-2011 NeXus International Advisory Committee (NIAC)
# 
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
# version 3 of the License, or (at your option) any later version.
#
# This library is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with this library; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
#
# For further information, see http://www.nexusformat.org

########### SVN repository information ###################
# $Date: 2011-01-26 23:49:00 +0000 (Wed, 26 Jan 2011) $
# $Author: Pete Jemian $
# $Revision: 808 $
# $HeadURL: file:///isis/svn/nexus/definitions/trunk/manual/examples/NX__template__.nxdl.xml $
# $Id: NX__template__.nxdl.xml 808 2011-01-26 23:49:00Z Pete Jemian $
########### SVN repository information ###################
-->
<definition name="NX__template__" extends="NXobject" type="group"
    category="application"
    xmlns="http://definition.nexusformat.org/nxdl/3.1"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://definition.nexusformat.org/nxdl/3.1 ../nxdl.xsd"
    version="1.0b"
    >
    <doc>template for a NXDL application definition</doc>
</definition>

For example, copy and rename the file to NXwoni.nxdl.xml. Then, locate the XML root element definition and change the name attribute (the XML shorthand for this attribute is /definition/@name) to NXwoni. Change the doc as well. Also consider keeping track of /definition/@version as suits your development of this NXDL file.

The next thing which needs to be done is adding groups into the definition. A group is defined by some XML like this:

<group type="NXdata">

</group>

The type is the actual NeXus base class this group belongs to. Optionally a name attribute may be given (default is data).

Next, one needs to include data items too. The XML for such a data item looks like this:

<field name="polar_angle" type="NX_FLOAT units="NX_ANGLE">
  <doc>Link to polar angle in /NXentry/NXinstrument/NXdetector</doc>
  <dimensions rank="1">
    <dim index="1" value="ndet"/>
  </dimensions>
</field>

The meaning of the name attribute is intuitive, the type can be looked up in the relevant base class definition. A field definition can optionally contain a doc element which contains a description of the data item. The dimensions entry specifies the dimensions of the data set. The size attribute in the dimensions tag sets the rank of the data, in this example: rank="1". In the dimensions group there must be rank dim fields. Each dim tag holds two attributes: index determines to which dimension this tag belongs, the 1 means the first dimension. The value attribute then describes the size of the dimension. These can be plain integers, variables, such as in the example ndet or even expressions like tof+1.

Thus a NXDL file can be constructed. The full NXDL file for the WONI example is given in the section called “Full listing of the WONI Application Definition”. Clever readers may have noticed the strong similarity between our working example NXwoni and NXmonopd since they are essentially identical. Give yourselves a cookie if you spotted this.

Basically you are done. Your first application definition for NeXus is constructed. In order to make your work a standard for that particular application type, some more steps are required:

The NIAC must review an application definition before it is accepted as a standard. The one year curation period is in place in order to gain practical experience with the definition and to sort out bugs from Step 1. In this period, data shall be written and analyzed using the new application definition.

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="nxdlformat.xsl" ?>
<!--
# NeXus - Neutron and X-ray Common Data Format
# 
# Copyright (C) 2008-2011 NeXus International Advisory Committee (NIAC)
# 
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
# version 3 of the License, or (at your option) any later version.
#
# This library is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with this library; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
#
# For further information, see http://www.nexusformat.org

########### SVN repository information ###################
# $Date: 2011-02-04 16:02:51 +0000 (Fri, 04 Feb 2011) $
# $Author: Pete Jemian $
# $Revision: 811 $
# $HeadURL: file:///isis/svn/nexus/definitions/trunk/applications/NXmonopd.nxdl.xml $
# $Id: NXmonopd.nxdl.xml 811 2011-02-04 16:02:51Z Pete Jemian $
########### SVN repository information ###################
-->
<definition name="NXmonopd" extends="NXobject" type="group"
    category="application"
    xmlns="http://definition.nexusformat.org/nxdl/3.1"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://definition.nexusformat.org/nxdl/3.1 ../nxdl.xsd"
    version="1.0b"
    svnid="$Id: NXmonopd.nxdl.xml 811 2011-02-04 16:02:51Z Pete Jemian $">
    <doc> Monochromatic Neutron and X-Ray Powder Diffraction. Instrument definition for a powder
        diffractometer at a monochromatic neutron or X-ray beam. This is both suited for a powder
        diffractometer with a single detector or a powder diffractometer with a position sensitive
        detector. </doc>
    <group type="NXentry" name="entry">
        <field name="title"/>
        <field name="start_time" type="NX_DATE_TIME"/>
        <field name="definition">
            <doc> Official NeXus NXDL schema to which this file conforms </doc>
            <enumeration>
                <item value="NXmonopd"/>
            </enumeration>
        </field>
        <group type="NXinstrument">
            <group type="NXsource">
                <field name="type"/>
                <field name="name"/>
                <field name="probe">
                    <enumeration>
                        <item value="neutron"/>
                        <item value="x-ray"/>
                        <item value="electron"/>
                    </enumeration>
                </field>
            </group>
            <group type="NXcrystal">
                <field name="wavelength" type="NX_FLOAT" units="NX_WAVELENGTH">
                    <doc>Optimum diffracted wavelength</doc>
                    <dimensions rank="1">
                        <dim index="1" value="i"/>
                    </dimensions>
                </field>
            </group>
            <group type="NXdetector">
                <field name="polar_angle" type="NX_FLOAT" axis="1">
                  <doc>where ndet = number of detectors</doc>
                  <dimensions rank="1">
                    <dim index="1" value="ndet" />
                  </dimensions>
                </field>
                <field name="data" type="NX_INT" signal="1">
                  <doc>
                    detector signal (usually counts) are already
                    corrected for detector efficiency
                  </doc>
                  <dimensions rank="1">
                    <dim index="1" value="ndet" />
                  </dimensions>
                </field>
            </group>
        </group>
        <group type="NXsample">
            <field name="name">
                <doc>Descriptive name of sample</doc>
            </field>
            <field name="rotation_angle" type="NX_FLOAT" units="NX_ANGLE">
                <doc> Optional rotation angle for the case when the powder diagram has been obtained
                    through an omega-2theta scan like from a traditional single detector powder
                    diffractometer </doc>
            </field>
        </group>
        <group type="NXmonitor">
            <field name="mode">
                <doc>Count to a preset value based on either clock time (timer) or received monitor
                    counts (monitor).</doc>
                <enumeration>
                    <item value="monitor"/>
                    <item value="timer"/>
                </enumeration>
            </field>
            <field name="preset" type="NX_FLOAT">
                <doc>preset value for time or monitor</doc>
            </field>
            <field name="integral" type="NX_FLOAT" units="NX_ANY">
                <doc>Total integral monitor counts</doc>
            </field>
        </group>
        <group type="NXdata">
            <link name="polar_angle" target="/NXentry/NXinstrument/NXdetector/polar_angle">
                <doc>Link to polar angle in /NXentry/NXinstrument/NXdetector</doc>
            </link>
            <link name="data" target="/NXentry/NXinstrument/NXdetector/data">
                <doc>Link to data in /NXentry/NXinstrument/NXdetector</doc>
            </link>
        </group>
    </group>
</definition>

The application definition is like an interface for your data file. In practice files will contain far more information. For this, the extendable capability of NeXus comes in handy. More data can be added, and upstream software relying on the interface defined by the application definition can still retrieve the necessary information without any changes to their code.

NeXus application definitions only standardize classes. You are free to decide upon names of groups, subject to them matching regular expression for NeXus name attributes (see the regular expression pattern for NXDL group and field names in #RegExpName). Note the length limit of 63 characters imposed by HDF5. Please use sensible, descriptive names and separate multi worded names with underscores.

Something most people wish to add is more metadata, for example in order to index files into a database of some sort. Go ahead, do so, if applicable, scan the NeXus base classes for standardized names. For metadata, consider to use the NXarchive definition. In this context, it is worth to mention that a practical NeXus file might adhere to more then one application definition. For example, WONI data files may adhere to both the NXmonopd and NXarchive definitions. The first for data analysis, the second for indexing into the database.

Often, instrument scientists want to store the complete state of their instrument in data files in order to be able to find out what went wrong if the data is unsatisfactory. Go ahead, do so, please use names from the NeXus base classes.

Site policy might require you to store the names of all your bosses up to the current head of state in data files. Go ahead, add as many NXuser classes as required to store that information. Knock yourselves silly over this.

Your Scientific Accounting Department (SAD) may ask of you the preposterous; to store billing information into data files. Go ahead, do so if your judgment allows. Just do not expect the NIAC to provide base classes for this and do not use the prefix NX for your classes.

In most cases, NeXus files will just have one NXentry class group. But it may be required to store multiple related data sets of the results of data analysis into the same data file. In this case create more entries. Each entry should be interpretable standalone, i.e. contain all the information of a complete NXentry class. Please keep in mind that groups or data items which stay constant across entries can always be linked in.

To update files in these repositories you will need to use a subversion client such as TortoiseSVN/^[22] for Microsoft Windows or svn for command-line shells and also provide your NeXus Wiki username and password. Note that for subversion write access:

Here are the URLs to access the subversion repositories as a developer:

Please report any problems via the Issue Reporting system.

As well as needing a valid account, you will not be able to check-in changes unless you indicate (in the log message attached to the commit) which current issues on the Issue Reporting system the changes either fix or refer to. This is done by enclosing special phrases in the commit message of the form:

command #1
command #1, #2
command #1 & #2
command #1 and #2

where command is one of the commands detailed below and #1 means issue number 1 on the system, etc. You can have more then one command in a message. The following commands are supported and there is more then one spelling for each command (to make this as user-friendly as possible):

For example, the commit message

Changed blah and foo to do this or that. Fixes #10 and #12, and refs #12.

This will close issues #10 and #12, and add a note to #12 on the Issue Reporting system. For a list of current issues, see:

Many Uniform Resource Locators (URLs) have been used in this section. This is a table describing them.