I am glad to hear that more people is interested in reverse-engineering NiGHTS games! n_n
I would like to share my speculations as someone who has attempted reverse engineering the Saturn game (and a little bit of the modern port), I'm pretty sure that you (and many other members of this community) already know this, but it might be helpful to recall and document publicly for further researches:
- The PS2/PS3/360/PC ports alltogether are most likely based off a completely different base engine compared to Saturn NiGHTS. Which is an understandable decision considering that remaking the whole game into a modern brand new engine with platform portability features (and preexistent assets) is cheaper and convenient, even in the long term (they would be able to keep porting the game in the future, if they wanted!). They really did a good job recreating the essential heuristics of the original NiGHTS game (even with the noticeable amount of discreppancies)!
- Apparently, Saturn NiGHTS engine wasn't created with porting features in mind at all, and the engine itself was absolutely bounded to the internals of SEGA Saturn. Plus, all the components of the legacy development environment (DOS/Windows 3.1-95/Mac OS 7-8 tools) that they might have used to program the game are possibly stored somewhere in SEGA, and were shared with the porting team, but weren't practical to fully reutilize, as they possibly required to be refurbished and reverse engineered to make them run properly on modern operating systems (it's not impossible, but when developing a port it requires a lot of time and knowledge of programmers that don't work on SEGA anymore, at least in the extreme case that there is not enough official documentation of NiGHTS Engine tools' source code). So, ultimately, they proably used a portion of the runnable tools to view and extract game assets directly. It was a pretty flexible and scalable engine though! As it was used in Sonic Jam and Burning Rangers, so in fact, newer versions of the engine existed at some point.
- In addition to the last paragraph, the original source code was most likely written in C/C++ with portions of SH2 (and other saturn chipsets) assembly code. In consequense, the models, graphics, music, and binary data in general, is compressed and stored in a specific way that can be conveniently interpreted and represented by an engine that is deeply connected with Sega Saturn's harware.
Is it possible to create a new executable that uses saturn files? Yes, you can potentially create an executable application that behaves like the original game and loads the original binary files by mapping the code that makes saturn's hardware calls into another operating system's calls. But rather than investing resources in such research, it was cheaper for SEGA to rip/convert data and make a new engine from scratch (or simply emulate it, in order to play the original one in other platforms).
- The new engine was most likely written in C/C++ and uses a more standard way of storing and representing data. In consequence, all the modern ports use almost all the same internal files, but for each platform, a different executable and runtime environment that invokes them. Windows port makes heavy use of DirectX 9 to run the game (a standard library that is also very well documented).
- Speaking of formats, the modern ports use streamed .ADX music instead of sequenced music with soundfonts. Graphics were converted to a modern format that is also compatible with more diverse runtime environments. Since ADX is a common format used in SEGA, it's possible that many of the other formats used in the new port were already reverse-engineered by other SEGA fans. If you don't find the exact file extension documented anywhere, it's also posible that even if the extension is different, the file is an already known format used in SEGA but with a slight variation.
- You can also research common formats used in PS2 games, it's still useful considering that, if I remember correctly, the newer ports for PC/PS3/360 still have remnats of common format files used in PS2 games.
- Additionaly, it will be a lot easier to reverse-engineer the PC port, as you can use plenty of debugging tools directly with the binary application.
I personally have had more luck modifying portions of the original Saturn game as I am more used to debug it with emulator features, but I am lightyears away from internalizing and undertanding SH2 assembly code, so I will probably move to the Windows port somewhere in the future.
With that all in mind, I wish you much luck into this task. And don't give up! Because at the end of the day you will still learn a LOT, even if reverse engineering NiGHTS is a very specific thing, just keep dreaming.