Hi cees,
I'm not an expert on libusb but I don't think you should mix libusb and libusb-1.0. If you decide to use libusb-1.0 you should install libusb-1.0 and link with -lusb-1.0, and also copy libusb-1.0 from the rpi. Make sure you copy the real so-file because so-files are often links to more specific version so-files. And you may have to create new link files with 'ln -s'.
By running the cross compiler with -v you can see which library paths the compiler uses. You can also add new paths with -L, but there should not be any path specified in -l.
/Claes