Some time ago I needed to link with ARM DSP library on NuttX to use FFT feature and I decided to document who I did it and some tricks to reduce the final binary size.
In fact to include a library all you need to do is add it to your board Make.defs:
EXTRA_LIBS += lname
Where lname is your libname.
But after the compilation you will notice that your nuttx.bin binary will become very big. It happens because the linker will include all the functions in the library.
You can instruct it to include only the needed function using this parameter with LDFLAGS:
LDFLAGS += --gc-sections
Now your binary will become way smaller