I just ported NuttX to Atmel MEGA1284P-XPLAINED board using as base the moteino-mega board that is very similar. Main difference is the 16MHz crystal that became 11.0592MHz in the MEGA1284P-XPLAINED.

These are the steps to compile NuttX:

1) Install the AVR GCC toolchain:

$ sudo apt install gcc-avr avr-libc

2) Configure NuttX to compile to this board:

$ ./tools/configure.sh mega1284p-xplained:nsh
$ make

3) Flash the generated nuttx.hex file:

$ sudo avrdude -p atmega1284p -c avr910 -P /dev/ttyACM0 -b57600 -F -u -U flash:w:nuttx.hex:i

For some strange reason it is failing:

avrdude: avr910_recv(): programmer is not responding
Found programmer: Id = "AVRBOOT"; type = S
    Software Version = 1.5; Hardware Version = ?.
Programmer supports auto addr increment.
Programmer supports buffered memory access with buffersize = 256 bytes.

Programmer supports the following devices:

avrdude: warning: selected device is not supported by programmer: m1284p
avrdude: error: programmer did not respond to command: select device
avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.09s

avrdude: Device signature = 0x97050d
avrdude: Expected signature for ATmega1284P is 1E 97 05
avrdude: NOTE: "flash" memory has been specified, an erase cycle will be performed
         To disable this feature, specify the -D option.
avrdude: erasing chip
avrdude: error: programmer did not respond to command: chip erase
avrdude: reading input file "nuttx.hex"
avrdude: writing flash (55260 bytes):

Writing | ################################################## | 100% 23.64s

avrdude: 55260 bytes of flash written
avrdude: verifying flash memory against nuttx.hex:
avrdude: load data flash data from input file nuttx.hex:
avrdude: input file nuttx.hex contains 55260 bytes
avrdude: reading on-chip flash data:

Reading
...
Reading | ################################################## | 100% 10.89s

avrdude: verifying ...
avrdude: verification error, first mismatch at byte 0x0000
         0x0d != 0x0c
avrdude: verification error; content mismatch
avrdude: error: programmer did not respond to command: leave prog mode

avrdude done.  Thank you.

So, it is not working yet, but we are near!!! Stay tuned!